+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20220527/A0c --output-run DEEP/20220527/A0c/science#step1/20240319T010033Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.9049 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0069 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0154 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/pipeline_orig.qgraph DEEP/20220527/A0c/science#step1/20240319T010033Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.4711 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 14.5129 seconds; current memory usage: 0.363 Gibyte, delta: 0.185 Gibyte, peak delta: 0.173 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.363 Gibyte (main), 0.356 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0204 seconds; current memory usage: 0.363 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.363 Gibyte (main), 0.356 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 744 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20220527_A0c_science#step1_20240319T010033Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.1514 seconds; current memory usage: 0.364 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.364 Gibyte (main), 0.356 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 746 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.7227 seconds; current memory usage: 0.365 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.365 Gibyte (main), 0.356 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 7c7c3bc5-fe94-47c4-adc6-67ae771ac048 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-19 01:01:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-19 01:01:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-19 01:01:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-19 01:01:21 proc_lsst.multi:170 [INFO] [multi] job_id 25873 proc_lsst.multi INFO: [multi] job_id 25873 2024-03-19 01:01:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:21 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 25873 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 25873 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 25873 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 5dcce012d36c3233f5fbc91e454861b9 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162074860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/216bc129-743a-48b0-bdaa-20131e6bfa7f_isr_1101469_41.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/216bc129-743a-48b0-bdaa-20131e6bfa7f_isr_1101469_41.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 1ac3a038e4aa4e8d7a32d8b4418f78b7 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162075120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0c28a6c9-9499-478b-a54b-d77c623e1bad_isr_1101472_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0c28a6c9-9499-478b-a54b-d77c623e1bad_isr_1101472_40.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash a77f6b2f06718f128c6e5db92d77ec55 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620753a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/9fdd1b75-6f06-40eb-8634-e78107c94894_isr_1101466_14.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/9fdd1b75-6f06-40eb-8634-e78107c94894_isr_1101466_14.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 060120feb0fe5bfb1281a78353404483 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162075580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/f21088fb-738d-4b7e-a8f6-aaa1b96253ef_isr_1101475_52.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/f21088fb-738d-4b7e-a8f6-aaa1b96253ef_isr_1101475_52.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 285e548c40a49b58487e177de55b865b parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162075a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/edcb0cb2-cf49-4ac1-934a-7302c86d61da_isr_1101475_13.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/edcb0cb2-cf49-4ac1-934a-7302c86d61da_isr_1101475_13.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 2f3dd9714b56b32c594172f1de59c38e parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620760c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d90f2de1-a08c-44b3-9c16-86ef039c9a53_isr_1101472_15.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d90f2de1-a08c-44b3-9c16-86ef039c9a53_isr_1101472_15.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 3613e043ed4c424ab3471a81509b39a7 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162076200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a0c913fe-5618-4cf8-8c9e-c2d76bde504c_isr_1101472_36.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a0c913fe-5618-4cf8-8c9e-c2d76bde504c_isr_1101472_36.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 2c563fa4258154903994c1bfee26f1c9 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162076480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/6b54e84f-421c-4a64-98cf-34d2d7a474fa_isr_1101469_17.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/6b54e84f-421c-4a64-98cf-34d2d7a474fa_isr_1101469_17.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 4bee86952e7384edfa3611e4d524e307 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620777e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/c5ad0679-1f9b-401a-b376-d7391f5eff31_isr_1101466_61.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/c5ad0679-1f9b-401a-b376-d7391f5eff31_isr_1101466_61.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash f2c69614a67cc80b6644ffa9df26f8f3 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620774c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/c93f2502-9df9-4411-8381-d72b51589f7d_isr_1101469_5.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/c93f2502-9df9-4411-8381-d72b51589f7d_isr_1101469_5.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash f45c84fb6c23d9ab8a2815db96f0a8ea parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620771a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/883c156f-820d-4003-8a5d-b5005c329001_isr_1101475_25.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/883c156f-820d-4003-8a5d-b5005c329001_isr_1101475_25.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 62861f678f08201d6bcc9521d358610e parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162076e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/e9f509f8-2e3b-4e74-8d95-b0a1fcf58467_isr_1101466_60.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/e9f509f8-2e3b-4e74-8d95-b0a1fcf58467_isr_1101466_60.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 15b4d5bd0752fd4ee8cde14aae1cc2e5 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162076b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/b97df112-bd01-472a-b1aa-67ad92d8cec8_isr_1101469_38.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/b97df112-bd01-472a-b1aa-67ad92d8cec8_isr_1101469_38.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 495fa0cf9d1f2cdf0110fda6a76f9a03 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/b03fceca-34bf-4c06-9f89-1815b147dfed_isr_1101469_50.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/b03fceca-34bf-4c06-9f89-1815b147dfed_isr_1101469_50.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 862f3443cee0aac780139fb14b7483d9 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162077ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/3f709873-2353-4b28-a12c-1d988c30befb_isr_1101472_49.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/3f709873-2353-4b28-a12c-1d988c30befb_isr_1101472_49.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 637cbdac4f0a68db099e4ba3014a6110 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162077ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/ebebec09-57bd-4e87-9672-7198b25ef3de_isr_1101472_31.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/ebebec09-57bd-4e87-9672-7198b25ef3de_isr_1101472_31.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash e1dfcef87c8ebb51970d6b181368339f parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162098220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/79352830-9bde-426a-bfda-833a12cf94e2_isr_1101475_45.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/79352830-9bde-426a-bfda-833a12cf94e2_isr_1101475_45.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 525772b6095d2670a5d9a2b033e39560 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162098540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/20d45f7b-c1a7-49de-bd1f-1a7c0014215b_isr_1101472_24.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/20d45f7b-c1a7-49de-bd1f-1a7c0014215b_isr_1101472_24.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash bbc555ae225415e6dd2099e168f17952 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162098860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/4e522a7d-935f-4a81-83ce-6992e7f88c3f_isr_1101472_18.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/4e522a7d-935f-4a81-83ce-6992e7f88c3f_isr_1101472_18.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 0229b327635e6ccac8b847078e5dc288 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162098b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/f0695a6a-abb0-4e89-a2f6-5cbbfd4a03db_isr_1101475_60.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/f0695a6a-abb0-4e89-a2f6-5cbbfd4a03db_isr_1101475_60.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 4cb185c7427c0c399c667ad75fc67ca5 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162098ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/ee21e3fc-1cc3-49b6-927f-d719de56f27f_isr_1101475_35.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/ee21e3fc-1cc3-49b6-927f-d719de56f27f_isr_1101475_35.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 8409c1dc68c70760db7d41925d54ad62 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620991c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/d730b5da-2a9c-4966-8e9a-df581e647515_isr_1101469_47.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/d730b5da-2a9c-4966-8e9a-df581e647515_isr_1101469_47.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash dfbaeb8411ce1f1c31542e2d8109c4d7 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620994e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c838db30-7d10-4a8b-ae11-9fbb79963bf6_isr_1101475_51.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c838db30-7d10-4a8b-ae11-9fbb79963bf6_isr_1101475_51.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 9fb3291da622319dc9e3aaba5c966572 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/cf06c4e0-cda1-4e18-96bf-cc861ffa2793_isr_1101475_16.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/cf06c4e0-cda1-4e18-96bf-cc861ffa2793_isr_1101475_16.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 592dfb18142f25d889404a2d34178b57 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/f06826c7-8a02-4741-9614-b903b7946e49_isr_1101472_39.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/f06826c7-8a02-4741-9614-b903b7946e49_isr_1101472_39.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 7ade1f4f96edbcc63975e0cd51e23aa8 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/4ec9eab0-0777-4d82-ac07-c8c910274468_isr_1101469_7.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/4ec9eab0-0777-4d82-ac07-c8c910274468_isr_1101469_7.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 3e8ce8e4543973177b946f70076da8af parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/910bad8c-b87c-4b6d-b0a6-14278b1d7d05_isr_1101469_61.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/910bad8c-b87c-4b6d-b0a6-14278b1d7d05_isr_1101469_61.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 6a94688c3ce86c1298b948daf736f410 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9bf29db4-dc51-4a74-842c-0bd76bccfcd8_isr_1101469_11.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9bf29db4-dc51-4a74-842c-0bd76bccfcd8_isr_1101469_11.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 261a687247cff155d551725d145f405a parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/be20ba6e-dacc-488f-9408-143ee0d96bf9_isr_1101475_24.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/be20ba6e-dacc-488f-9408-143ee0d96bf9_isr_1101475_24.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash ac0967e0aa25cdeea605c2acd57c0954 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a787bad6-0143-401d-be21-6c0f842f5f8e_isr_1101475_27.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a787bad6-0143-401d-be21-6c0f842f5f8e_isr_1101475_27.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 72cecce7eafeefa15de4f30207ed1021 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/fd3b038e-eb67-4d39-9c65-6626fac1aea6_isr_1101469_15.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/fd3b038e-eb67-4d39-9c65-6626fac1aea6_isr_1101469_15.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 1b8805cc94431968d258f70bac3ade5b parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/d5bb8a7c-3abb-4a4b-865b-9a0765173e2a_isr_1101466_30.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/d5bb8a7c-3abb-4a4b-865b-9a0765173e2a_isr_1101466_30.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 221e1dda3ae44d376261da4543dec132 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c3b2650f-a76a-46ca-9e1c-47765660423a_isr_1101475_5.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c3b2650f-a76a-46ca-9e1c-47765660423a_isr_1101475_5.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 8a17359d64882b506631fdba4c7d4fbe parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/82d432a8-86d4-432a-b2ac-087a00cb475c_isr_1101472_48.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/82d432a8-86d4-432a-b2ac-087a00cb475c_isr_1101472_48.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 965e7f18f4c65092b8d884562d86c033 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b852a8c1-7c07-47f2-9d9d-a6a8ee9e111e_isr_1101466_25.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b852a8c1-7c07-47f2-9d9d-a6a8ee9e111e_isr_1101466_25.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash d34f324a24d1f5a8f04230e79427e41a parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/ff5b76d0-4715-4405-b0b4-4f519495d874_isr_1101475_1.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/ff5b76d0-4715-4405-b0b4-4f519495d874_isr_1101475_1.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 0130825765f1a67eaf7422a9cb0bf861 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/174953cb-73af-4d17-9c86-889af3377824_isr_1101469_48.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/174953cb-73af-4d17-9c86-889af3377824_isr_1101469_48.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 8895304d2070f19849e3beda0b5840f7 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/60152752-4214-4230-ac04-ceddfde70f54_isr_1101466_53.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/60152752-4214-4230-ac04-ceddfde70f54_isr_1101466_53.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 2ec956f819543eda17848ef2858b1722 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/66148eca-ca32-4955-8b97-4d67177efe3a_isr_1101472_14.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/66148eca-ca32-4955-8b97-4d67177efe3a_isr_1101472_14.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash aa4abf574160bd3f825c451a5d1e5273 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/f753dd15-fa99-4a80-b8d4-092080524cc2_isr_1101469_14.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/f753dd15-fa99-4a80-b8d4-092080524cc2_isr_1101469_14.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 9380fb67cdc2876fa8322fe3501f138a parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/2f037e97-5b49-4c82-8721-ca8c56ed0524_isr_1101466_34.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/2f037e97-5b49-4c82-8721-ca8c56ed0524_isr_1101466_34.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash d23bad82b7dff13f184e7903a8f0a0e4 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/47ff6b20-8235-4c5c-add3-48c0470d7654_isr_1101472_41.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/47ff6b20-8235-4c5c-add3-48c0470d7654_isr_1101472_41.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 1cb4f1ad9b97a56ddceb1fd553f4b527 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/5ceba667-89a7-4d96-be98-164a8bd8848e_isr_1101469_60.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/5ceba667-89a7-4d96-be98-164a8bd8848e_isr_1101469_60.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash fc52b7c041b61b9d1d7099b65bceb702 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/baa6773c-5bee-47ee-9200-b434aceed1c3_isr_1101472_51.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/baa6773c-5bee-47ee-9200-b434aceed1c3_isr_1101472_51.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 25f0ce7decb222e21d66b7cb50629c2a parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/1f8b4bca-f081-48bf-8500-d4dea2fa9b71_isr_1101469_12.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/1f8b4bca-f081-48bf-8500-d4dea2fa9b71_isr_1101469_12.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 2a0923a4491cb2587f24e0e7ec1e2412 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bdd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/8f9e8ad0-3fb4-4e50-92b5-517fac91e6e2_isr_1101466_12.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/8f9e8ad0-3fb4-4e50-92b5-517fac91e6e2_isr_1101466_12.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 4ed0b5bf13447d78a63ac57c7b6e8d14 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620be020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/0eca8887-346f-4a62-83da-cd9b5746ce7a_isr_1101466_41.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/0eca8887-346f-4a62-83da-cd9b5746ce7a_isr_1101466_41.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 7b2cab659e6d9b78223856e5ea5fe2e6 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620be340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9d5844cf-dff2-4f70-97d2-ba0df7a8e5e8_isr_1101469_37.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9d5844cf-dff2-4f70-97d2-ba0df7a8e5e8_isr_1101469_37.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 83fe9efe813a179c7b2f2354ae0716c5 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620be660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/65d2968d-c993-4b59-90dc-c05f9622a6b5_isr_1101466_2.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/65d2968d-c993-4b59-90dc-c05f9622a6b5_isr_1101466_2.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 76d85694a8377991dc519cf238947e1a parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620be980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/fe6c6fd5-33fc-4b1a-a09b-cbf5d384e08e_isr_1101469_40.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/fe6c6fd5-33fc-4b1a-a09b-cbf5d384e08e_isr_1101469_40.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 6ffa7f14ac9550dea15ccd10adf7b914 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620beca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/6644c07c-681e-4821-971d-7b486b73f881_isr_1101469_32.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/6644c07c-681e-4821-971d-7b486b73f881_isr_1101469_32.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 5cd88c9f2e953749d7e1468792cf94fb parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620befc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/b4dac980-e2af-4e4c-a99f-72063e4db717_isr_1101475_61.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/b4dac980-e2af-4e4c-a99f-72063e4db717_isr_1101475_61.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash fe87653938016b710eb9d0682060e6c3 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/9f39b649-1cba-4bfb-bc7f-cf967bf571a8_isr_1101475_14.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/9f39b649-1cba-4bfb-bc7f-cf967bf571a8_isr_1101475_14.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 856205f449dc0d040cc5765ed01ba897 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/1627d05b-238f-4787-96fc-c084dfc5e577_isr_1101466_27.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/1627d05b-238f-4787-96fc-c084dfc5e577_isr_1101466_27.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 8613d9815377283956e49df3f5b9457c parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/902c7e5d-0f3c-413a-822c-f8914dd6afc5_isr_1101469_20.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/902c7e5d-0f3c-413a-822c-f8914dd6afc5_isr_1101469_20.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 22c1957abb99d6af37a718a66e780e3f parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/75373f35-b63f-42a3-88db-42e9d3fd9489_isr_1101475_50.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/75373f35-b63f-42a3-88db-42e9d3fd9489_isr_1101475_50.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 78a9c8941791beafe72e00cd86f16819 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/85acd5cf-1227-4f17-989b-90ba8e95e0e9_isr_1101466_1.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/85acd5cf-1227-4f17-989b-90ba8e95e0e9_isr_1101466_1.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 86287a58e7df3a8b009334174aa68a97 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/92503d1f-0854-4384-8b87-8af4b5b29592_isr_1101469_57.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/92503d1f-0854-4384-8b87-8af4b5b29592_isr_1101469_57.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 322eba8583b624a92993c63ed855cbfb parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/7dfd929e-2b52-49e7-a176-457157262a80_isr_1101466_18.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/7dfd929e-2b52-49e7-a176-457157262a80_isr_1101466_18.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 825078ac95eac8929c65c5cb49fc574a parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a928af09-9daa-4a04-9d86-d3ad8a3a3191_isr_1101475_44.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a928af09-9daa-4a04-9d86-d3ad8a3a3191_isr_1101475_44.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash d71fbc7e8320ae673f7505d4cfed6092 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/24afbb0f-fa54-4388-8917-718b8cb96d9d_isr_1101472_61.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/24afbb0f-fa54-4388-8917-718b8cb96d9d_isr_1101472_61.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash eab71dd05f39676fa04d6099ff799d8c parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/87b6f434-d5aa-481d-9fe5-f67b6a6368de_isr_1101472_1.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/87b6f434-d5aa-481d-9fe5-f67b6a6368de_isr_1101472_1.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 69dede63ddd1174f494c448925c0e240 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/abcdc947-c607-42b2-9074-2f0e23e0c432_isr_1101466_17.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/abcdc947-c607-42b2-9074-2f0e23e0c432_isr_1101466_17.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash d788c8d7e092b124b6edf2d0c142af23 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/e1403547-6cf5-48bd-bc95-b8aa579d83e2_isr_1101475_29.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/e1403547-6cf5-48bd-bc95-b8aa579d83e2_isr_1101475_29.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 137bab43adee9cb802dec1e604d68510 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/f1284538-f04e-4432-b289-3971fab52846_isr_1101466_16.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/f1284538-f04e-4432-b289-3971fab52846_isr_1101466_16.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 4f7b226a61534258b5ff472c744377ee parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1b14e5a4-b5fa-4d59-ac92-020988d48749_isr_1101472_3.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1b14e5a4-b5fa-4d59-ac92-020988d48749_isr_1101472_3.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 645dddd92f2ffe59ecd087c41a144528 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/451646ff-f31c-4783-8358-09a53327cdfc_isr_1101466_57.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/451646ff-f31c-4783-8358-09a53327cdfc_isr_1101466_57.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 63f1ead9a6a553725f428f0767de3bdb parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/1f0a726a-a272-4870-825a-74be5c8ca6a5_isr_1101469_34.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/1f0a726a-a272-4870-825a-74be5c8ca6a5_isr_1101469_34.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 64ca7c9c857523ffa0356315f0c64300 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/3831aa3c-916b-4598-82f8-a236a8139562_isr_1101475_48.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/3831aa3c-916b-4598-82f8-a236a8139562_isr_1101475_48.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 13df7afbcd766e7abb7b226ce7076b87 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/83fa3b67-6079-4b07-9b38-0e51704b49da_isr_1101475_12.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/83fa3b67-6079-4b07-9b38-0e51704b49da_isr_1101475_12.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 77117e7e7172b967e83e8bdf78abee05 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/6b9d2ba9-b619-41ef-95c8-ebc93097cab2_isr_1101472_6.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/6b9d2ba9-b619-41ef-95c8-ebc93097cab2_isr_1101472_6.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 75f71a82ea747ab7e4ef42c21ba64cef parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/f3da9084-9a7f-43a5-b665-b3233d46c4e4_isr_1101475_41.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/f3da9084-9a7f-43a5-b665-b3233d46c4e4_isr_1101475_41.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash c0b7250037fa047856c2e7a4a4ad9e8f parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/e1486017-5913-4a92-80e2-c58842513d85_isr_1101466_6.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/e1486017-5913-4a92-80e2-c58842513d85_isr_1101466_6.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 057bfc91831296426f7640aefbca1827 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/bba5d219-d04e-4fd4-813a-ff12d5d43380_isr_1101475_31.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/bba5d219-d04e-4fd4-813a-ff12d5d43380_isr_1101475_31.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 87c93e320ecdbeb5143ad63bbd869f17 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1e8c108d-03e1-4a1b-839d-88027c642972_isr_1101472_58.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1e8c108d-03e1-4a1b-839d-88027c642972_isr_1101472_58.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash e904cdb09cfd5460141ad2d557280eb2 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/0256731b-82c3-4768-a405-ef6d36100719_isr_1101475_53.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/0256731b-82c3-4768-a405-ef6d36100719_isr_1101475_53.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash f4f07381ad204ac36f34f4d7c2347432 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/a1ea8027-f43f-41ce-8575-1c8a230a63c2_isr_1101469_51.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/a1ea8027-f43f-41ce-8575-1c8a230a63c2_isr_1101469_51.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash b859da5ff4348deffcfd351229179b2e parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/8c296908-2e32-4cfb-bfbb-0ba0c6728262_isr_1101475_22.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/8c296908-2e32-4cfb-bfbb-0ba0c6728262_isr_1101475_22.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash e452e36e7c87aa8ec5c28cb60d4c073d parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/d2652ce8-aeeb-4bcc-8c2d-85b9a7869bcf_isr_1101475_54.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/d2652ce8-aeeb-4bcc-8c2d-85b9a7869bcf_isr_1101475_54.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 3ea0105206a9f69d631ff1fe35f2010b parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/e33c16f2-1616-42b0-bb87-13ea4a2c14a9_isr_1101469_33.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/e33c16f2-1616-42b0-bb87-13ea4a2c14a9_isr_1101469_33.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 07b717f5a67bc23f4bd1d572ef3d6e7a parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/85c64e87-924a-4364-921c-bb36c48ebf26_isr_1101472_52.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/85c64e87-924a-4364-921c-bb36c48ebf26_isr_1101472_52.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash d54de62e72d1c7056fb172316ed4e1fe parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/07bf8b1c-53b6-41ee-a5e9-48bf3fb47b0c_isr_1101469_27.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/07bf8b1c-53b6-41ee-a5e9-48bf3fb47b0c_isr_1101469_27.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash d2dcba502073c637a18f7d5272ff4b57 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160add120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/577469ee-3f05-42fa-a5da-f724b53212a1_isr_1101472_43.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/577469ee-3f05-42fa-a5da-f724b53212a1_isr_1101472_43.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 7f70fb7f68989b173ecf221f5795a7d1 parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160add440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/bae8b176-f066-4c97-ab5b-130dc77b25a3_isr_1101472_4.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/bae8b176-f066-4c97-ab5b-130dc77b25a3_isr_1101472_4.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash b403dc2be1c0b69b63728b7e08c7bb1c parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160add760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/3aa254e1-30ed-4598-b0ab-fe2c763885bb_isr_1101475_28.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/3aa254e1-30ed-4598-b0ab-fe2c763885bb_isr_1101475_28.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash f7fa8b95b733ff36dfaea16770aa325d parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/ac276ecc-c8ed-4563-9932-01075f450750_isr_1101472_56.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/ac276ecc-c8ed-4563-9932-01075f450750_isr_1101472_56.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash f0c1b6a26b6b912e0b9c9617a7b6c220 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160addda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/5797468c-3f54-4a22-8a76-cb815eb74973_isr_1101466_37.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/5797468c-3f54-4a22-8a76-cb815eb74973_isr_1101466_37.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash f4d5cb767f8cc880be98710e60f934e6 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ade0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/65b55b3d-e736-47b5-8c29-2197ebe5fcd9_isr_1101466_49.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/65b55b3d-e736-47b5-8c29-2197ebe5fcd9_isr_1101466_49.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash b6c619b5115f5df702f4837be8424a9c parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ade3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c4b2906b-b320-4542-9034-bfc83bc17072_isr_1101475_9.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c4b2906b-b320-4542-9034-bfc83bc17072_isr_1101475_9.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 9b3ce8ca43958b354fc2eaab163226f2 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ade700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9328e366-e0ba-4ef8-aff6-0720c4d04ad4_isr_1101469_62.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9328e366-e0ba-4ef8-aff6-0720c4d04ad4_isr_1101469_62.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 172a7b9c2c50a936decd80b6b6da9db1 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0c3d90a1-c094-4f6d-afa4-04563fecef50_isr_1101469_13.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0c3d90a1-c094-4f6d-afa4-04563fecef50_isr_1101469_13.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 8187cc8182fe5a890d7fcdf2170dbec9 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160aded40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/daa158f2-4f1d-48ec-b2c9-bf44d73db330_isr_1101472_23.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/daa158f2-4f1d-48ec-b2c9-bf44d73db330_isr_1101472_23.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 8acc4ba14562cd01858f27684dc21fb9 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/04bef5aa-f921-4bd6-8144-dcae6dc02c37_isr_1101469_3.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/04bef5aa-f921-4bd6-8144-dcae6dc02c37_isr_1101469_3.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 4b68c5ce72a57a12b0b390dbe96470ca parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/59276905-783a-4806-82aa-b2aa0ea86aa4_isr_1101475_40.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/59276905-783a-4806-82aa-b2aa0ea86aa4_isr_1101475_40.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash e009b887f1d6fefc8518cd1d9a8b589a parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0854749c-4e80-4c69-8bf4-6d22532817ad_isr_1101469_8.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0854749c-4e80-4c69-8bf4-6d22532817ad_isr_1101469_8.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash dd10e464b694d4b66675de2d2e9427b9 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/29a8fd3b-dfdf-4d88-9e0e-dbeacd906ec8_isr_1101475_18.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/29a8fd3b-dfdf-4d88-9e0e-dbeacd906ec8_isr_1101475_18.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 37a34ed0a45758e63d701aaa773d2a5b parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9e3ff556-e6b0-46bf-a7a9-0e7dc587f4f8_isr_1101469_19.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9e3ff556-e6b0-46bf-a7a9-0e7dc587f4f8_isr_1101469_19.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 51efae0d916c381240cad3739267f880 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b04040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/b5bbf232-1902-4d62-a9c4-321531cab7db_isr_1101475_17.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/b5bbf232-1902-4d62-a9c4-321531cab7db_isr_1101475_17.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash bff7d1bfdb01b3baa84fa23044121e94 parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b04360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0759c5ad-d155-4c25-8ded-27a17abcda06_isr_1101469_2.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0759c5ad-d155-4c25-8ded-27a17abcda06_isr_1101469_2.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash a50873897920141a934af6e590844154 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b04680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/6d7f8d49-6c35-4024-815f-cd14a75161ff_isr_1101466_51.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/6d7f8d49-6c35-4024-815f-cd14a75161ff_isr_1101466_51.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 4ef6868147653757c73d8f315bc98a39 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b049a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/f0ad033b-5650-4a4a-9616-aac1a904d8b1_isr_1101472_11.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/f0ad033b-5650-4a4a-9616-aac1a904d8b1_isr_1101472_11.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 9a744435acd9efc655e6366326385f71 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b04cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0fb5ea05-6e6a-425e-9f72-5eb0d7827856_isr_1101472_28.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0fb5ea05-6e6a-425e-9f72-5eb0d7827856_isr_1101472_28.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash cebcb76ab8548e3818c1689fd0de30c0 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b04fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/91de3da8-a06c-4aeb-9ad2-9959faea33d0_isr_1101475_58.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/91de3da8-a06c-4aeb-9ad2-9959faea33d0_isr_1101475_58.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 28d31e901f1c0fade5c557ac56a0fdd4 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/71f1f39d-8384-4c3a-b287-4cf780910166_isr_1101469_30.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/71f1f39d-8384-4c3a-b287-4cf780910166_isr_1101469_30.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 4ae4062f19fcc4a6ed25fa0f5c50d9dc parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/fad05104-f53a-4749-a3c2-e169b74c18d4_isr_1101466_3.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/fad05104-f53a-4749-a3c2-e169b74c18d4_isr_1101466_3.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 6b95dd54e612c039dc06afa7cb33d683 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/fc973936-98f3-40b2-ba0c-edddd5dd758f_isr_1101475_36.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/fc973936-98f3-40b2-ba0c-edddd5dd758f_isr_1101475_36.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 31a02de03caa6c4fbb02c3b73eee134e parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/64ed4f09-8598-4126-963d-c5ac991c7a1c_isr_1101475_26.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/64ed4f09-8598-4126-963d-c5ac991c7a1c_isr_1101475_26.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 49c30c6c7703720eb4717d3b4358f3a0 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d5b5d17f-f8fd-48e3-8ca3-697d0bf05c1a_isr_1101472_16.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d5b5d17f-f8fd-48e3-8ca3-697d0bf05c1a_isr_1101472_16.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash a91de09c0d3189103c35e990fb510206 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b062a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/98cb4190-3ef7-42d7-9a36-8d32a5310c92_isr_1101469_18.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/98cb4190-3ef7-42d7-9a36-8d32a5310c92_isr_1101469_18.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 18aa59901520c799913a013f2729d603 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b065c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/362cfe0b-b65f-458b-b263-ed70df027f1b_isr_1101475_6.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/362cfe0b-b65f-458b-b263-ed70df027f1b_isr_1101475_6.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 2dcf33eb27747675d5d7f1dba73aecf3 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b068e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/8f566d6f-eceb-485f-9ec9-0296ce48286e_isr_1101466_50.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/8f566d6f-eceb-485f-9ec9-0296ce48286e_isr_1101466_50.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash a31f7eced81abe7f4fd5d321af13ec87 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b06c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/d81e4646-5f52-438c-b141-b0cb35b8bef4_isr_1101469_46.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/d81e4646-5f52-438c-b141-b0cb35b8bef4_isr_1101469_46.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 77dd0af6e235a28b92ae2b1755366b95 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b06f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/6bf99484-a427-4fc6-8242-4c6c39037549_isr_1101472_17.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/6bf99484-a427-4fc6-8242-4c6c39037549_isr_1101472_17.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 080f3d8b4e2797e20989b4128e9c79f6 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b07240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/dcee3b48-3d24-4e69-9a29-e4be9c4c082d_isr_1101469_25.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/dcee3b48-3d24-4e69-9a29-e4be9c4c082d_isr_1101469_25.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash f707811dc812c7a190b35b93b9c0a7be parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b07560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0e7d7b54-1118-47f5-9355-32cb3a24cc0c_isr_1101472_13.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0e7d7b54-1118-47f5-9355-32cb3a24cc0c_isr_1101472_13.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 46b606fdce5a1bc41ec691d0eff22697 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b07880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/e3709366-d659-498d-8d2a-8c53c10be719_isr_1101469_43.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/e3709366-d659-498d-8d2a-8c53c10be719_isr_1101469_43.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash e5625ac272714e4627daec94bc3cce3e parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b07ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/94a64feb-25ec-47da-b524-eef58b7274c5_isr_1101472_33.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/94a64feb-25ec-47da-b524-eef58b7274c5_isr_1101472_33.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash a71df94fc9ddd0fe6742739f51babcfe parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b07ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/c0868192-8826-4c81-8766-1539d547ff8f_isr_1101469_58.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/c0868192-8826-4c81-8766-1539d547ff8f_isr_1101469_58.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 8e83c53e1623642ff9314ff429b0dc16 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b38220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d8c44fc5-fc95-4176-9edd-ad10052c17ef_isr_1101472_62.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d8c44fc5-fc95-4176-9edd-ad10052c17ef_isr_1101472_62.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash d7be9a659483b5a7dd61c9416b017b35 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b38540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/84b95863-06a1-4cf2-8c6e-6b7ae453fc2b_isr_1101469_9.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/84b95863-06a1-4cf2-8c6e-6b7ae453fc2b_isr_1101469_9.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash c709fbf4c3f3bb8f5c4bc0c2554b9ddf parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b38860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/42fcf718-45b1-432d-b9bf-9d87a9299a71_isr_1101475_57.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/42fcf718-45b1-432d-b9bf-9d87a9299a71_isr_1101475_57.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash b1064999a9c6dc783e5f927e5b802661 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b38b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0ac8b262-d06c-43c8-b6c7-c6a3177105b2_isr_1101472_42.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0ac8b262-d06c-43c8-b6c7-c6a3177105b2_isr_1101472_42.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 4ceadbdbfeb509c4482b289c8c7e843b parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b38ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/7a9452f6-e1a8-4c17-90fb-c1520a6a9321_isr_1101475_38.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/7a9452f6-e1a8-4c17-90fb-c1520a6a9321_isr_1101475_38.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 71d35fb29a46939e37c662edafb3e7ca parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b391c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/e131cf5c-effc-43d4-9f28-96482a528d28_isr_1101475_49.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/e131cf5c-effc-43d4-9f28-96482a528d28_isr_1101475_49.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 1cc7bb08c0751e2c8f1ab90e5d0f31ca parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b394e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/34f86c3b-3e29-419c-9dbd-b07e4d202f1a_isr_1101475_21.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/34f86c3b-3e29-419c-9dbd-b07e4d202f1a_isr_1101475_21.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash 54e18926c48fb556d739e4e773054f20 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b39800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-19 01:01:22 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/13fa7b39-4750-4a77-8250-66fb28c14055_isr_1101466_42.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/13fa7b39-4750-4a77-8250-66fb28c14055_isr_1101466_42.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 22846a6b25b421f462a9a88572556a92 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0e7b2c7a-b73e-4cbc-8fde-7bcdc34bfe31_isr_1101469_45.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0e7b2c7a-b73e-4cbc-8fde-7bcdc34bfe31_isr_1101469_45.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 2f53d1db92b2129d87da89574095cc80 parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b39b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0cf69265-f21a-47f3-9d2a-d71354aab47e_isr_1101469_23.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0cf69265-f21a-47f3-9d2a-d71354aab47e_isr_1101469_23.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash e320154a72a2826c62a9b57134713b28 parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b39ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/65091aaa-5810-4d01-8d40-9b47b1a3c3fd_isr_1101475_47.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/65091aaa-5810-4d01-8d40-9b47b1a3c3fd_isr_1101475_47.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 72c26d280ee954c796da58d8c67fcc37 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 129 parsl.dataflow.strategy DEBUG: Executor multi has 129 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 32 more blocks parsl.executors.status_handling INFO: Scaling out by 32 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-19 01:01:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-19 01:01:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-19 01:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/61902384-c23d-421e-afa9-bab7ed828b1a_isr_1101466_46.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/61902384-c23d-421e-afa9-bab7ed828b1a_isr_1101466_46.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash 2bddd70368fd6e4efdc9d27c0bdb4a7d parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/eb4c1013-34d9-46d1-a287-170f7ad61565_isr_1101466_35.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/eb4c1013-34d9-46d1-a287-170f7ad61565_isr_1101466_35.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash f9e191e662d8e5997b7d727a4c29a4cf parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/bce8bcd2-8825-4208-b6a4-2427c7008bb8_isr_1101472_54.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/bce8bcd2-8825-4208-b6a4-2427c7008bb8_isr_1101472_54.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash dad88048df0e77c585d0567c7db8446e parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/5825e854-9428-40eb-bc9e-230f73714cc0_isr_1101472_12.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/5825e854-9428-40eb-bc9e-230f73714cc0_isr_1101472_12.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash f35695329f493f2912474ab8330cf319 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/4644947b-288c-4dc6-bbe9-84ae3e8e15ed_isr_1101466_19.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/4644947b-288c-4dc6-bbe9-84ae3e8e15ed_isr_1101466_19.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 00ad592d28f4b4d2639bf76fab03cddb parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/83c5883d-c370-4019-97d9-66c32afa7349_isr_1101472_19.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/83c5883d-c370-4019-97d9-66c32afa7349_isr_1101472_19.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash eac311c294742d64ce9935411eb8d0a1 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b42e2bff-5c2d-498b-9978-2566a326afaf_isr_1101466_52.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b42e2bff-5c2d-498b-9978-2566a326afaf_isr_1101466_52.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash ee351e1aaa2246b8257db97763138c3c parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/0ba448ae-f915-4146-902f-790bf9bcb204_isr_1101466_21.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/0ba448ae-f915-4146-902f-790bf9bcb204_isr_1101466_21.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash e582b7decac27e580a2b2a7734688440 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b77380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/d1465afb-35cd-4b3a-b0bf-999006a86a40_isr_1101466_56.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/d1465afb-35cd-4b3a-b0bf-999006a86a40_isr_1101466_56.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 657c35f87c8f0462623445cb5a5b8b02 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b77060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/61bdb2ac-1127-49c0-9c31-bcf34c952631_isr_1101469_59.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/61bdb2ac-1127-49c0-9c31-bcf34c952631_isr_1101469_59.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash b755cad5ffc0669373c7f5da56f65bc4 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b76d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/9f7a509f-9388-429c-a19a-56da11502747_isr_1101466_28.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/9f7a509f-9388-429c-a19a-56da11502747_isr_1101466_28.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash 817802d99416aaeef551f0f3cfbb3571 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b76a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/7b496468-b7ec-4c65-8e8e-ced7c9efa109_isr_1101466_10.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/7b496468-b7ec-4c65-8e8e-ced7c9efa109_isr_1101466_10.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash ce14ad537c5f56bedb4c282bd1b764ee parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b76700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/bfdebce8-4fc3-4fbe-b5c5-a745ceb1763d_isr_1101466_11.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/bfdebce8-4fc3-4fbe-b5c5-a745ceb1763d_isr_1101466_11.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash 10aa176d75fde3013a8288bc836018f6 parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b763e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/259c10a0-5e48-4f4e-b0b3-a92a8390a1f1_isr_1101472_9.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/259c10a0-5e48-4f4e-b0b3-a92a8390a1f1_isr_1101472_9.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 9c6b534a550bf8388ef1a27d4b98bcff parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b74360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/2cb85398-5e88-4dac-be16-da478721bf03_isr_1101475_62.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/2cb85398-5e88-4dac-be16-da478721bf03_isr_1101475_62.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash d01211542f413e5dcf82aca637c5bfe6 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b745e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/5c3416cc-7d7f-4355-86f3-2f1bf67441c0_isr_1101472_59.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/5c3416cc-7d7f-4355-86f3-2f1bf67441c0_isr_1101472_59.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash d72ce0a0aebcccf0009f775804de3f5d parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b74900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/ac1b6702-dc04-434b-89ef-b898c525620a_isr_1101466_24.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/ac1b6702-dc04-434b-89ef-b898c525620a_isr_1101466_24.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash f62fffca0e4bf84d742f738f10adc9ac parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b74a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/469a2791-dbee-4471-8295-c02999033101_isr_1101466_4.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/469a2791-dbee-4471-8295-c02999033101_isr_1101466_4.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash f7011b4b30f1e7362a71c5beed7938b7 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b74d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/070adc68-5037-4fe8-86b3-2d7216fe7566_isr_1101472_29.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/070adc68-5037-4fe8-86b3-2d7216fe7566_isr_1101472_29.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash f730a70cdc1bb9c08e368fe157f3eab0 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b751c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/6d3c01ba-a09e-42e3-bc63-54467ba64581_isr_1101475_37.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/6d3c01ba-a09e-42e3-bc63-54467ba64581_isr_1101475_37.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 77a78ff2a74b08f5ec7c16083c388cd1 parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/12e92dfd-da82-447b-a099-0fddb6f928da_isr_1101469_22.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/12e92dfd-da82-447b-a099-0fddb6f928da_isr_1101469_22.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash d1f44a75afe4ecb8b3cde144a762b52a parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b75760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/76c16be2-77c5-4641-9f41-f35c9dbd429b_isr_1101475_33.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/76c16be2-77c5-4641-9f41-f35c9dbd429b_isr_1101475_33.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 80f7fac2111600037f16b2e12090e851 parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b75c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/e45a0eb0-ee20-4cd3-8ef8-843b33dd6df0_isr_1101472_60.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/e45a0eb0-ee20-4cd3-8ef8-843b33dd6df0_isr_1101472_60.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash 82b2a39015876ce19d122d094df7214c parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b75ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/10dd77ef-005e-492d-b0a0-53d5d599235c_isr_1101472_46.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/10dd77ef-005e-492d-b0a0-53d5d599235c_isr_1101472_46.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash c2cf5da8066c3d5a682e223ad2d11599 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b760c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/2e4d2491-3492-4f50-9710-d67738342247_isr_1101466_26.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/2e4d2491-3492-4f50-9710-d67738342247_isr_1101466_26.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 49b7d22df1684b3d8864589efec13f5b parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b776a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/c8394b16-9c40-488f-81ec-513b56672030_isr_1101472_22.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/c8394b16-9c40-488f-81ec-513b56672030_isr_1101472_22.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash 0a192573e228427645f821eecfbb01bc parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b779c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/66f547e2-2cf4-4dc1-8256-bf96639266bb_isr_1101472_38.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/66f547e2-2cf4-4dc1-8256-bf96639266bb_isr_1101472_38.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 706986fb37c34165ea070be6a82d624e parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b77ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/c83b8df3-dafa-4bd7-b78e-025493131df2_isr_1101469_52.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/c83b8df3-dafa-4bd7-b78e-025493131df2_isr_1101469_52.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash f6c3b670011bbeb62009fedf66ddb0cd parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/08459fe1-d266-4b84-ad60-3e688a5b60bf_isr_1101466_48.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/08459fe1-d266-4b84-ad60-3e688a5b60bf_isr_1101466_48.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash a0566bb4fc5befd3879aa8a25b19d430 parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/15f98f60-3bce-490a-9ff6-1f5fcba30498_isr_1101469_56.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/15f98f60-3bce-490a-9ff6-1f5fcba30498_isr_1101469_56.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash bfd0103f8a078a2ea1f738bf80c9323d parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-19 01:01:22 proc_lsst.multi:170 [INFO] [multi] job_id 17226175 proc_lsst.multi INFO: [multi] job_id 17226175 2024-03-19 01:01:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:22 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226175 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226175 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17226175 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-19 01:01:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-19 01:01:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/456ffadb-8b96-432a-bd43-7d38d1ac638d_isr_1101475_46.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/456ffadb-8b96-432a-bd43-7d38d1ac638d_isr_1101475_46.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash b490ba8c33b9c3e42f182ae3f9001472 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/98df82ef-c3e3-4b5e-8e3a-96e2ecaf71f8_isr_1101466_47.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/98df82ef-c3e3-4b5e-8e3a-96e2ecaf71f8_isr_1101466_47.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash eb7ef51dd02b948e4c31d12afdaf58b0 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/f28c1b42-2f3a-4ce4-b237-514b12a7f5b3_isr_1101466_62.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/f28c1b42-2f3a-4ce4-b237-514b12a7f5b3_isr_1101466_62.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash 8502eebde89b9c42585bdd850eca7bd5 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/25046fa8-f321-4893-8970-f9035bbcc078_isr_1101466_22.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/25046fa8-f321-4893-8970-f9035bbcc078_isr_1101466_22.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 2a775c076432c84a4c4efa81e3c29158 parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/4dcacf51-f388-43a8-bfb2-aa492bf5f7c8_isr_1101466_31.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/4dcacf51-f388-43a8-bfb2-aa492bf5f7c8_isr_1101466_31.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 17efa2f352b3841c46cdb2349b1834d9 parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c8f2ca01-94a7-4439-a491-43a3b20cdb24_isr_1101475_3.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c8f2ca01-94a7-4439-a491-43a3b20cdb24_isr_1101475_3.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 5a581c508d1c64756832735d0de847a3 parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/c43793a9-4b8e-4ec4-92b1-bdf080cea6cf_isr_1101466_44.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/c43793a9-4b8e-4ec4-92b1-bdf080cea6cf_isr_1101466_44.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 6a28b9446ed1ae96b3c41735cdb3bd3d parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bf740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/519a1485-20f9-4022-96cf-6dde2538fc08_isr_1101472_8.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/519a1485-20f9-4022-96cf-6dde2538fc08_isr_1101472_8.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash ca0f828e826ebd783e68ae27b51def43 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/286056e5-48c8-4def-a3ee-3f45aa3825b1_isr_1101469_44.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/286056e5-48c8-4def-a3ee-3f45aa3825b1_isr_1101469_44.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash 92dfc163d09179c0ba19cd2752d63452 parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/eb32b3a5-cf01-410f-b60a-00b2e7a00c83_isr_1101475_39.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/eb32b3a5-cf01-410f-b60a-00b2e7a00c83_isr_1101475_39.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 626a4471b4889a8c2d80209093a2ccd7 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/01624b05-ad30-475b-aa5e-49828a90c265_isr_1101475_19.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/01624b05-ad30-475b-aa5e-49828a90c265_isr_1101475_19.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 0e21ea3d7936b20b0deb4b6b2a781bad parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/048d1933-a116-43ff-ab59-3bd8bcabfd50_isr_1101466_54.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/048d1933-a116-43ff-ab59-3bd8bcabfd50_isr_1101466_54.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash d2dd32d7e07366168ffc09c9667f59b4 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609be0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1275757b-3246-445c-bb09-a1d841c84a6c_isr_1101472_32.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1275757b-3246-445c-bb09-a1d841c84a6c_isr_1101472_32.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 221bae2aca17d7d79cd4e079361c632a parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609be520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1b12dab2-3c5f-418a-b13a-cabedb66f10c_isr_1101472_55.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1b12dab2-3c5f-418a-b13a-cabedb66f10c_isr_1101472_55.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash d0832f6aa68713bd558a5d816ca64d92 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609be840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/54e68ab2-258e-4fc8-bbb2-107a3a342710_isr_1101466_20.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/54e68ab2-258e-4fc8-bbb2-107a3a342710_isr_1101466_20.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 839fea963d7b364e0f568cacf8f41cb2 parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609beac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/acd8c3b1-9e6f-4964-b6ef-6983b246a21c_isr_1101466_55.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/acd8c3b1-9e6f-4964-b6ef-6983b246a21c_isr_1101466_55.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 39250b82dc14c6c0e3b94adcf13c33fd parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609befc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/fa40109a-69a8-44e3-96d4-842f8e180e17_isr_1101472_20.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/fa40109a-69a8-44e3-96d4-842f8e180e17_isr_1101472_20.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash f683cefc93e4f654890a25fe4473b1bf parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/d223ba77-1db0-4fa2-8848-22cfb1fcc02d_isr_1101475_59.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/d223ba77-1db0-4fa2-8848-22cfb1fcc02d_isr_1101475_59.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 9c0fece2925b7660adacd98fcf0bef7d parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/01b759bc-77b1-4c4e-8efd-e309af887303_isr_1101466_5.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/01b759bc-77b1-4c4e-8efd-e309af887303_isr_1101466_5.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 35530129eaf5753ffc3c1b63272e2355 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/6e0859f2-56a7-492d-9035-453410656bf2_isr_1101472_27.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/6e0859f2-56a7-492d-9035-453410656bf2_isr_1101472_27.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash 709e11be58faf3a37039a4b8c7f3e0a8 parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/5c74439b-3bfc-4428-bf6b-68ee62d68c2c_isr_1101475_2.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/5c74439b-3bfc-4428-bf6b-68ee62d68c2c_isr_1101475_2.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 64aab6190e8b3612bf6fb5cba3927643 parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/cd05aa6c-5a88-4472-a6e7-2a985dfe51d5_isr_1101475_11.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/cd05aa6c-5a88-4472-a6e7-2a985dfe51d5_isr_1101475_11.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 5e852fd1a42282263c1b9f940570bf56 parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a51960a8-9f54-4f2b-83ce-5bdbd1c1f418_isr_1101475_8.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a51960a8-9f54-4f2b-83ce-5bdbd1c1f418_isr_1101475_8.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash e7b4f3c3a557a9d5ae209175b0047f23 parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/b3c0d116-926a-4040-9302-00163234cb7e_isr_1101469_42.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/b3c0d116-926a-4040-9302-00163234cb7e_isr_1101469_42.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 8e7269fba3a763c02122c5d8032821e3 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a051c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/fbb45ccd-7463-4d40-8fe4-1f7311f47b7e_isr_1101475_56.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/fbb45ccd-7463-4d40-8fe4-1f7311f47b7e_isr_1101475_56.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 3e17ed9f540ae0799124c5c98da2d439 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a054e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/cfa31fbc-0513-46e6-a014-a74dfb93265e_isr_1101469_31.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/cfa31fbc-0513-46e6-a014-a74dfb93265e_isr_1101469_31.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash 2fe3bf02608f2e0335464562407c1b06 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a05800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-19 01:01:22 proc_lsst.multi:170 [INFO] [multi] job_id 17226176 proc_lsst.multi INFO: [multi] job_id 17226176 2024-03-19 01:01:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 01:01:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:22 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226176 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226176 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17226176 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-19 01:01:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-19 01:01:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 01:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a765c1cb-9c4e-4b13-9864-63daaddc3502_isr_1101475_32.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a765c1cb-9c4e-4b13-9864-63daaddc3502_isr_1101475_32.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash 8181a6a7fb45f8e32d6c05c9eb7d7bce parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a058a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/2b9a3330-0593-423f-9c71-b0f56d875bc3_isr_1101472_30.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/2b9a3330-0593-423f-9c71-b0f56d875bc3_isr_1101472_30.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558 parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559 parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash 29fd6301767642f53a9c19e20a4a8397 parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a05d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/e631e2b3-00c1-4fac-8d57-4f2bffc6f83d_isr_1101469_55.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/e631e2b3-00c1-4fac-8d57-4f2bffc6f83d_isr_1101469_55.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561 parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562 parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 503b25ac5e487a8bbd51414869e9374f parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a06020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/698167f8-167c-40c2-947c-f4c1a2175fac_isr_1101475_43.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/698167f8-167c-40c2-947c-f4c1a2175fac_isr_1101475_43.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564 parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565 parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 50cf4de739d876dfe8adedf4d3a98171 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/72c75c1f-e77b-4eca-a52d-c4a768226772_isr_1101469_35.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/72c75c1f-e77b-4eca-a52d-c4a768226772_isr_1101469_35.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567 parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568 parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 8a4237febe04b02bec5705d689f4a1b9 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a06480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/666722b0-5ff7-40af-83d0-444b7e601058_isr_1101466_40.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/666722b0-5ff7-40af-83d0-444b7e601058_isr_1101466_40.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570 parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571 parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 8100f27a4087e7833945a810bc2d0545 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a06840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/753d9d3c-8ab5-44d5-b2a5-19b778e95730_isr_1101472_50.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/753d9d3c-8ab5-44d5-b2a5-19b778e95730_isr_1101472_50.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573 parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574 parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash c3fdbf53470d9c46b916a1b470747111 parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a06b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/95197869-5337-406c-a112-b085d9f3f284_isr_1101472_35.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/95197869-5337-406c-a112-b085d9f3f284_isr_1101472_35.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576 parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577 parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash 2698b69d4eb6cbcbbcf240166646136d parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a06e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/26a5f816-ea71-4dfb-8675-d647330d362e_isr_1101472_21.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/26a5f816-ea71-4dfb-8675-d647330d362e_isr_1101472_21.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579 parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580 parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 7352bc6385222680cb91b0cd62031100 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/d4c2a2b6-c193-4e37-921b-e507fefb2c34_isr_1101475_23.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/d4c2a2b6-c193-4e37-921b-e507fefb2c34_isr_1101475_23.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582 parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583 parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 4034ff455b35a7436af517ddfd65e5ca parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/86ce974f-54c1-42bc-8f17-a821b6dd9031_isr_1101466_59.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/86ce974f-54c1-42bc-8f17-a821b6dd9031_isr_1101466_59.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585 parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586 parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash dc78e1cbf1c0c2704e0e78fb1d559b24 parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a076a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/6bae0d28-e020-4754-a418-968f259d4b7d_isr_1101469_54.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/6bae0d28-e020-4754-a418-968f259d4b7d_isr_1101469_54.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588 parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589 parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 551c6ee8240504a97078a8336d42e33f parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/cd601cbd-078a-4258-9203-c6afa0dc6355_isr_1101466_23.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/cd601cbd-078a-4258-9203-c6afa0dc6355_isr_1101466_23.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591 parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592 parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 79292d2552ff6016fbbd0853e3c5f352 parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/9fa98f43-9d08-4610-9c3f-a2217b54c5ab_isr_1101475_15.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/9fa98f43-9d08-4610-9c3f-a2217b54c5ab_isr_1101475_15.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594 parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595 parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash 7460f28ebd9c275cefa9a9b7fcb95b77 parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/46dd967e-a4a9-43ae-a0ff-0430602fa0d7_isr_1101469_21.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/46dd967e-a4a9-43ae-a0ff-0430602fa0d7_isr_1101469_21.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597 parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598 parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 4f7adb8a02fa9c26def6a058081e903c parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/59d33119-980b-4a32-87c3-ed98a9939cb7_isr_1101469_49.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/59d33119-980b-4a32-87c3-ed98a9939cb7_isr_1101469_49.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600 parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601 parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash f768548ff7e3dec2c6f684eb376c29e5 parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b1315bb7-4d18-4233-9db2-01c4f96a6c99_isr_1101466_13.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b1315bb7-4d18-4233-9db2-01c4f96a6c99_isr_1101466_13.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603 parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604 parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash 6f76f3b23315c0454ea5ce9e2e9a2f18 parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b37f485a-6f98-4103-826e-4cae6c601e1b_isr_1101466_36.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b37f485a-6f98-4103-826e-4cae6c601e1b_isr_1101466_36.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606 parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607 parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash 56ef12314aca0d3eccf11514936bed0d parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/4971ef5b-df21-4cca-87f8-d33a29a51fff_isr_1101472_2.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/4971ef5b-df21-4cca-87f8-d33a29a51fff_isr_1101472_2.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609 parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610 parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 1d9cd4da7af128b55d045ddbecd80916 parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/00ffac42-e0c4-477a-a4c5-81e49b2dea9d_isr_1101472_57.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/00ffac42-e0c4-477a-a4c5-81e49b2dea9d_isr_1101472_57.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612 parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613 parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash 4274ece609c02bee6805572c3bf90fa8 parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/bad96ae4-85c5-498a-9591-184f19a169f5_isr_1101466_38.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/bad96ae4-85c5-498a-9591-184f19a169f5_isr_1101466_38.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615 parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616 parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash 524eecb827dae471ba2bd19b125e689b parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/9cf9d04d-e85e-4009-b3e9-1d9580468e07_isr_1101475_20.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/9cf9d04d-e85e-4009-b3e9-1d9580468e07_isr_1101475_20.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618 parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619 parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 351cf9e39a042be05cbf2e848662cda3 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-19 01:01:23 proc_lsst.multi:170 [INFO] [multi] job_id 17226177 proc_lsst.multi INFO: [multi] job_id 17226177 2024-03-19 01:01:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 01:01:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226177 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226177 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17226177 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-19 01:01:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-19 01:01:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 01:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/01624a47-06dc-4f6f-a74b-4a8505a45aff_isr_1101469_10.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/01624a47-06dc-4f6f-a74b-4a8505a45aff_isr_1101469_10.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621 parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622 parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash addf053f379917d43d028fc9b849bd66 parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/cd40b11c-9f27-4b1b-8e9e-61e027cc7d73_isr_1101472_7.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/cd40b11c-9f27-4b1b-8e9e-61e027cc7d73_isr_1101472_7.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624 parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625 parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 26fbca2a20fe16e6005402b8141d97dc parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/8de921e0-2a90-4471-b1e1-df321e2d82d2_isr_1101472_10.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/8de921e0-2a90-4471-b1e1-df321e2d82d2_isr_1101472_10.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627 parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628 parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash f0d5108342bdedcea3335bbc26a13c35 parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a0b01fcd-46f5-4a35-ac83-3da25e2be4b0_isr_1101472_26.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a0b01fcd-46f5-4a35-ac83-3da25e2be4b0_isr_1101472_26.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630 parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631 parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash f638c2b0a678cf94e9f23601d384559b parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/73b55365-0b76-409a-806a-b8860df409a6_isr_1101469_36.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/73b55365-0b76-409a-806a-b8860df409a6_isr_1101469_36.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633 parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634 parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 897d80d1f9ec041528b8e4169321aede parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/4769ee8f-f69e-41c1-8a51-2f3729af687f_isr_1101475_30.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/4769ee8f-f69e-41c1-8a51-2f3729af687f_isr_1101475_30.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636 parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637 parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash 1006fb1efd45c9fefa7cf0125af3d65f parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a236a505-f60a-49f2-a89a-1001474786e5_isr_1101472_5.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a236a505-f60a-49f2-a89a-1001474786e5_isr_1101472_5.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639 parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640 parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash df1476da419bd59ab9b950038c8776a2 parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/3dd4d308-3015-4cf9-b641-1c878a747ec5_isr_1101472_37.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/3dd4d308-3015-4cf9-b641-1c878a747ec5_isr_1101472_37.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642 parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643 parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash 957c62b8bceba8d0a285d93555dc29b3 parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/04991cd6-7e68-4528-91c7-a7d2bdc83d7f_isr_1101475_10.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/04991cd6-7e68-4528-91c7-a7d2bdc83d7f_isr_1101475_10.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645 parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646 parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash e62ad23dc5add36891cd23280736ea96 parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/5b227a2b-b44d-4ae1-8085-18f3071277ac_isr_1101475_4.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/5b227a2b-b44d-4ae1-8085-18f3071277ac_isr_1101475_4.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648 parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649 parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash fb0569782cad957d211f395173206bef parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/03c4a7c3-5968-441e-b92b-a2b3688195e7_isr_1101472_25.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/03c4a7c3-5968-441e-b92b-a2b3688195e7_isr_1101472_25.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651 parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652 parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash e3cb84509a7966e21a72e3cd17d3754e parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/5cb2c4b1-182a-4563-8479-eb91d70f0baa_isr_1101469_1.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/5cb2c4b1-182a-4563-8479-eb91d70f0baa_isr_1101469_1.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654 parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655 parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash 228232d67eda0731a2682deaf421c469 parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/45256cd9-25a1-4585-a5f1-5701dd4baaaf_isr_1101466_7.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/45256cd9-25a1-4585-a5f1-5701dd4baaaf_isr_1101466_7.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657 parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658 parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash a7cad2914fe7fa5c1d9ced7701406c2f parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/8ba93521-acb0-409a-9320-b4c9286529f7_isr_1101469_4.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/8ba93521-acb0-409a-9320-b4c9286529f7_isr_1101469_4.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660 parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661 parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 981b531c6a41f2db8526d8df40ae9224 parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/3b4dc638-ad81-4362-bd3c-4daa667ee3cd_isr_1101466_8.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/3b4dc638-ad81-4362-bd3c-4daa667ee3cd_isr_1101466_8.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663 parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664 parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash bf842d590ef6b3949240886f54b4a401 parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9b2f46c0-3eec-4b1b-ab0a-c06f673f3dee_isr_1101469_39.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9b2f46c0-3eec-4b1b-ab0a-c06f673f3dee_isr_1101469_39.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666 parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667 parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash 5b2df008a95af81d9acb64d26fcd58c0 parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/2aeeb6b7-a80f-4190-b5c8-e6b445322a65_isr_1101466_32.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/2aeeb6b7-a80f-4190-b5c8-e6b445322a65_isr_1101466_32.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669 parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670 parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash 7ef3bfd4b22ea021e15329fc1ce7c1df parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/03ee38db-4b2c-4775-ae05-01ab3c42d354_isr_1101466_58.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/03ee38db-4b2c-4775-ae05-01ab3c42d354_isr_1101466_58.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672 parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673 parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash b24b0d915453a7fcf0819768658700dc parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1c2d8f73-a6ba-45a1-af63-b9e648f3d9f1_isr_1101472_34.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1c2d8f73-a6ba-45a1-af63-b9e648f3d9f1_isr_1101472_34.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675 parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676 parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash b942ab2e3444fcbcd8aff4b6b867c076 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/035f1484-9acf-434f-9f72-01b83708c48a_isr_1101469_26.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/035f1484-9acf-434f-9f72-01b83708c48a_isr_1101469_26.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678 parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679 parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash a159192188c7b2f6a53085cefbdd96fc parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/966b2fa6-52fa-404a-acd6-65df7c484ca3_isr_1101466_45.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/966b2fa6-52fa-404a-acd6-65df7c484ca3_isr_1101466_45.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681 parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682 parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash e62cbccb4f0b34262f44d700dc683b10 parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/54f9a8bd-1cca-41dd-8c30-5901cc2b1025_isr_1101466_29.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/54f9a8bd-1cca-41dd-8c30-5901cc2b1025_isr_1101466_29.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684 parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685 parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 7e50f341328f906b347895891d3a30d7 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/9f6ce7a8-0443-4688-8ce0-6eee6efac2bb_isr_1101472_44.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/9f6ce7a8-0443-4688-8ce0-6eee6efac2bb_isr_1101472_44.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687 parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688 parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash e09316219e7df10e8856ab281dacfc18 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/2f11d5fa-b2a8-4004-a14b-d9eee1a32efe_isr_1101469_24.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/2f11d5fa-b2a8-4004-a14b-d9eee1a32efe_isr_1101469_24.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690 parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691 parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash 44b5ebe4c2cebc2fd3d9b1f945aed234 parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c69aface-bd6c-4d63-a5b0-87771cbebbed_isr_1101475_7.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c69aface-bd6c-4d63-a5b0-87771cbebbed_isr_1101475_7.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693 parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694 parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash 6057c43be3f5c9fbe2c0a381a157d268 parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/bc63409f-263f-4899-8fc3-ea1dd5372245_isr_1101466_43.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/bc63409f-263f-4899-8fc3-ea1dd5372245_isr_1101466_43.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696 parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697 parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash 01171106e968f6cdcb4eb0c0ee0ab954 parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/887979dd-8f0f-47b9-bec2-8e38caaff152_isr_1101466_39.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/887979dd-8f0f-47b9-bec2-8e38caaff152_isr_1101466_39.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699 parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700 parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash b7b324ad1f592042beb61b3ace9a13dd parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/ce53ed7a-6e11-4124-960a-4ecef76dedf5_isr_1101469_16.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/ce53ed7a-6e11-4124-960a-4ecef76dedf5_isr_1101469_16.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702 parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703 parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash 99d23c64eb1fbd95d50e49a03c03078a parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/c6a820cb-4ab0-4c56-b966-6a5b39b56491_isr_1101466_9.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/c6a820cb-4ab0-4c56-b966-6a5b39b56491_isr_1101466_9.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705 parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706 parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash adca080a3e3b233b7730a32e9dd6df65 parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/ed7e42a5-26bc-4a90-ace2-44fdf553a2c5_isr_1101469_28.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/ed7e42a5-26bc-4a90-ace2-44fdf553a2c5_isr_1101469_28.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708 parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709 parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 48571c8386e23a240bef511ec0a91d60 parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-19 01:01:23 proc_lsst.multi:170 [INFO] [multi] job_id 17226178 proc_lsst.multi INFO: [multi] job_id 17226178 2024-03-19 01:01:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 01:01:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226178 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226178 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17226178 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-19 01:01:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-19 01:01:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 01:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/975f0522-8187-45e8-84ba-53dc24e5e024_isr_1101469_6.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/975f0522-8187-45e8-84ba-53dc24e5e024_isr_1101469_6.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711 parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712 parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 767fe7d83f6710fc3fbcb5ad1fc4dfa0 parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/57646193-b9c7-4531-8853-19cd9d940deb_isr_1101466_33.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/57646193-b9c7-4531-8853-19cd9d940deb_isr_1101466_33.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714 parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715 parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash ee2ae28b388fc4a2c5c94015a416987d parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a1134e9e-fd84-4197-8eaf-13d27b7932ab_isr_1101472_53.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a1134e9e-fd84-4197-8eaf-13d27b7932ab_isr_1101472_53.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717 parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718 parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash 452760e81ef82e7ee4a2250b4a74ae0c parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ec2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/dccf0fad-11b3-46db-9d5d-68bf8c851887_isr_1101469_29.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/dccf0fad-11b3-46db-9d5d-68bf8c851887_isr_1101469_29.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720 parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721 parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 01736a1a9fcdea8dcd947d083c1f7224 parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ec5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/957e324c-4716-4f42-9ea7-dab325f84d74_isr_1101469_53.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/957e324c-4716-4f42-9ea7-dab325f84d74_isr_1101469_53.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723 parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724 parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 8c88324af58a471a494a4e7d412fbb00 parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ec900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1113f154-dd41-4aab-a295-1747ae018250_isr_1101472_45.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1113f154-dd41-4aab-a295-1747ae018250_isr_1101472_45.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726 parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727 parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 734464c277850245484ac089d22c33ef parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ecc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d368f1d8-e041-43ba-be26-f9377fd189c0_isr_1101472_47.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d368f1d8-e041-43ba-be26-f9377fd189c0_isr_1101472_47.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729 parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730 parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash 22d23aa72a50cff4413db19a5a97f904 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608efd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/6de27fed-97fc-44e8-a665-217ea162886f_isr_1101475_42.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/6de27fed-97fc-44e8-a665-217ea162886f_isr_1101475_42.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732 parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733 parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash 2b2bfa48abd93f3050087682aec3068c parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608efa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/9756227c-16c9-44d5-bdfb-bbc25d125904_isr_1101466_15.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/9756227c-16c9-44d5-bdfb-bbc25d125904_isr_1101466_15.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735 parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736 parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 992b2ec9360bce2a2277c99e70d1a86a parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ef740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/52e68085-dde9-46b3-931e-b474fecab22f_isr_1101475_55.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/52e68085-dde9-46b3-931e-b474fecab22f_isr_1101475_55.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738 parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739 parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash 515e852524fd7b8c51c7b4bdc4afaeeb parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ed1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/74b6eea8-8010-4b86-b94c-b7c834b0eba8_isr_1101475_34.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/74b6eea8-8010-4b86-b94c-b7c834b0eba8_isr_1101475_34.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741 parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742 parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping 2024-03-19 01:01:23 proc_lsst.multi:170 [INFO] [multi] job_id 17226179 proc_lsst.multi INFO: [multi] job_id 17226179 2024-03-19 01:01:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226179 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226179 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17226179 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-19 01:01:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-19 01:01:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:23 proc_lsst.multi:170 [INFO] [multi] job_id 17226180 proc_lsst.multi INFO: [multi] job_id 17226180 2024-03-19 01:01:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-19 01:01:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226180 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226180 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17226180 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-19 01:01:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-19 01:01:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-19 01:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:24 proc_lsst.multi:170 [INFO] [multi] job_id 17226181 proc_lsst.multi INFO: [multi] job_id 17226181 2024-03-19 01:01:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-19 01:01:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226181 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226181 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17226181 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-19 01:01:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-19 01:01:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-19 01:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:24 proc_lsst.multi:170 [INFO] [multi] job_id 17226182 proc_lsst.multi INFO: [multi] job_id 17226182 2024-03-19 01:01:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-19 01:01:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226182 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226182 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17226182 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-19 01:01:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-19 01:01:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-19 01:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:24 proc_lsst.multi:170 [INFO] [multi] job_id 17226183 proc_lsst.multi INFO: [multi] job_id 17226183 2024-03-19 01:01:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-19 01:01:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226183 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226183 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17226183 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-19 01:01:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-19 01:01:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-19 01:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:24 proc_lsst.multi:170 [INFO] [multi] job_id 17226184 proc_lsst.multi INFO: [multi] job_id 17226184 2024-03-19 01:01:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-19 01:01:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226184 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226184 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17226184 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-19 01:01:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-19 01:01:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-19 01:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:25 proc_lsst.multi:170 [INFO] [multi] job_id 17226185 proc_lsst.multi INFO: [multi] job_id 17226185 2024-03-19 01:01:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-19 01:01:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226185 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226185 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17226185 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-19 01:01:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-19 01:01:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-19 01:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:25 proc_lsst.multi:170 [INFO] [multi] job_id 17226186 proc_lsst.multi INFO: [multi] job_id 17226186 2024-03-19 01:01:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-19 01:01:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226186 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226186 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17226186 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-19 01:01:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-19 01:01:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-19 01:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:25 proc_lsst.multi:170 [INFO] [multi] job_id 17226187 proc_lsst.multi INFO: [multi] job_id 17226187 2024-03-19 01:01:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-19 01:01:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226187 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226187 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17226187 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-19 01:01:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-19 01:01:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-19 01:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:25 proc_lsst.multi:170 [INFO] [multi] job_id 17226188 proc_lsst.multi INFO: [multi] job_id 17226188 2024-03-19 01:01:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-19 01:01:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226188 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226188 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17226188 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-19 01:01:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-19 01:01:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-19 01:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:26 proc_lsst.multi:170 [INFO] [multi] job_id 17226189 proc_lsst.multi INFO: [multi] job_id 17226189 2024-03-19 01:01:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-19 01:01:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226189 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226189 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17226189 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-19 01:01:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-19 01:01:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-19 01:01:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:26 proc_lsst.multi:170 [INFO] [multi] job_id 17226190 proc_lsst.multi INFO: [multi] job_id 17226190 2024-03-19 01:01:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-19 01:01:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226190 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226190 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17226190 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-19 01:01:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-19 01:01:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-19 01:01:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:26 proc_lsst.multi:170 [INFO] [multi] job_id 17226191 proc_lsst.multi INFO: [multi] job_id 17226191 2024-03-19 01:01:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-19 01:01:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226191 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226191 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17226191 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-19 01:01:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-19 01:01:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-19 01:01:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:27 proc_lsst.multi:170 [INFO] [multi] job_id 17226192 proc_lsst.multi INFO: [multi] job_id 17226192 2024-03-19 01:01:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-19 01:01:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226192 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226192 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17226192 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-19 01:01:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-19 01:01:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-19 01:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:27 proc_lsst.multi:170 [INFO] [multi] job_id 17226193 proc_lsst.multi INFO: [multi] job_id 17226193 2024-03-19 01:01:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-19 01:01:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226193 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226193 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17226193 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-19 01:01:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-19 01:01:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-19 01:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:27 proc_lsst.multi:170 [INFO] [multi] job_id 17226194 proc_lsst.multi INFO: [multi] job_id 17226194 2024-03-19 01:01:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 01:01:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226194 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226194 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17226194 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-19 01:01:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-19 01:01:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 01:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:28 proc_lsst.multi:170 [INFO] [multi] job_id 17226195 proc_lsst.multi INFO: [multi] job_id 17226195 2024-03-19 01:01:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-19 01:01:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226195 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226195 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17226195 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-19 01:01:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-19 01:01:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-19 01:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:28 proc_lsst.multi:170 [INFO] [multi] job_id 17226196 proc_lsst.multi INFO: [multi] job_id 17226196 2024-03-19 01:01:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-19 01:01:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226196 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226196 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17226196 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-19 01:01:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-19 01:01:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-19 01:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:28 proc_lsst.multi:170 [INFO] [multi] job_id 17226197 proc_lsst.multi INFO: [multi] job_id 17226197 2024-03-19 01:01:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-19 01:01:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226197 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226197 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17226197 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-19 01:01:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-19 01:01:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-19 01:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:28 proc_lsst.multi:170 [INFO] [multi] job_id 17226198 proc_lsst.multi INFO: [multi] job_id 17226198 2024-03-19 01:01:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-19 01:01:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226198 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226198 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17226198 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-19 01:01:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-19 01:01:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-19 01:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:28 proc_lsst.multi:170 [INFO] [multi] job_id 17226199 proc_lsst.multi INFO: [multi] job_id 17226199 2024-03-19 01:01:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-19 01:01:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226199 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226199 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17226199 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-19 01:01:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-19 01:01:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-19 01:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:29 proc_lsst.multi:170 [INFO] [multi] job_id 17226200 proc_lsst.multi INFO: [multi] job_id 17226200 2024-03-19 01:01:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-19 01:01:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:29 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226200 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226200 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17226200 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-19 01:01:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-19 01:01:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-19 01:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:29 proc_lsst.multi:170 [INFO] [multi] job_id 17226201 proc_lsst.multi INFO: [multi] job_id 17226201 2024-03-19 01:01:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-19 01:01:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:29 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226201 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226201 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17226201 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-19 01:01:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-19 01:01:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-19 01:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:30 proc_lsst.multi:170 [INFO] [multi] job_id 17226202 proc_lsst.multi INFO: [multi] job_id 17226202 2024-03-19 01:01:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-19 01:01:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:30 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226202 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226202 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17226202 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-19 01:01:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-19 01:01:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-19 01:01:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:31 proc_lsst.multi:170 [INFO] [multi] job_id 17226203 proc_lsst.multi INFO: [multi] job_id 17226203 2024-03-19 01:01:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-19 01:01:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:31 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226203 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226203 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17226203 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-19 01:01:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-19 01:01:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-19 01:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:31 proc_lsst.multi:170 [INFO] [multi] job_id 17226204 proc_lsst.multi INFO: [multi] job_id 17226204 2024-03-19 01:01:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:31 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17226204 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17226204 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17226204 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-19 01:01:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-19 01:01:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-19 01:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:32 proc_lsst.multi:170 [INFO] [multi] job_id 17226205 proc_lsst.multi INFO: [multi] job_id 17226205 2024-03-19 01:01:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:32 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17226205 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17226205 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17226205 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-19 01:01:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-19 01:01:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:33 proc_lsst.multi:170 [INFO] [multi] job_id 17226206 proc_lsst.multi INFO: [multi] job_id 17226206 2024-03-19 01:01:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 01:01:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:33 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17226206 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17226206 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17226206 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 33, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/32 running/pending blocks, and 18 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 33 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 29 more blocks parsl.executors.status_handling INFO: Scaling out by 29 blocks parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-19 01:01:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-19 01:01:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 01:01:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:35 proc_lsst.multi:170 [INFO] [multi] job_id 17226207 proc_lsst.multi INFO: [multi] job_id 17226207 2024-03-19 01:01:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 01:01:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:35 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17226207 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17226207 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17226207 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-19 01:01:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-19 01:01:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 01:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:36 proc_lsst.multi:170 [INFO] [multi] job_id 17226208 proc_lsst.multi INFO: [multi] job_id 17226208 2024-03-19 01:01:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 01:01:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:36 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17226208 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17226208 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17226208 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-19 01:01:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-19 01:01:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 01:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:37 proc_lsst.multi:170 [INFO] [multi] job_id 17226209 proc_lsst.multi INFO: [multi] job_id 17226209 2024-03-19 01:01:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:37 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17226209 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17226209 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17226209 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-19 01:01:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-19 01:01:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-19 01:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:39 proc_lsst.multi:170 [INFO] [multi] job_id 17226210 proc_lsst.multi INFO: [multi] job_id 17226210 2024-03-19 01:01:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226210 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226210 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17226210 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-19 01:01:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-19 01:01:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:40 proc_lsst.multi:170 [INFO] [multi] job_id 17226211 proc_lsst.multi INFO: [multi] job_id 17226211 2024-03-19 01:01:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 01:01:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:40 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226211 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226211 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17226211 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-19 01:01:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-19 01:01:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 01:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dcce012d36c3233f5fbc91e454861b9 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 6066f4f0fc789c4aa97157fc7f11bdb4 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162074ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/66d7c4d7-4a66-421c-92b9-e0e4621f5e51_characterizeImage_1101469_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/66d7c4d7-4a66-421c-92b9-e0e4621f5e51_characterizeImage_1101469_41.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/216bc129-743a-48b0-bdaa-20131e6bfa7f_isr_1101469_41.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/216bc129-743a-48b0-bdaa-20131e6bfa7f_isr_1101469_41.stderr 2024-03-19 01:01:41 proc_lsst.multi:170 [INFO] [multi] job_id 17226212 proc_lsst.multi INFO: [multi] job_id 17226212 2024-03-19 01:01:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 01:01:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:41 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226212 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226212 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17226212 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-19 01:01:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-19 01:01:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 01:01:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ac3a038e4aa4e8d7a32d8b4418f78b7 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 0c0133d460e6aadd1d5cec2cde8cd27e parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162074fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/07bb2f70-306d-4098-a07f-cf2a707754fa_characterizeImage_1101472_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/07bb2f70-306d-4098-a07f-cf2a707754fa_characterizeImage_1101472_40.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0c28a6c9-9499-478b-a54b-d77c623e1bad_isr_1101472_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0c28a6c9-9499-478b-a54b-d77c623e1bad_isr_1101472_40.stderr 2024-03-19 01:01:42 proc_lsst.multi:170 [INFO] [multi] job_id 17226213 proc_lsst.multi INFO: [multi] job_id 17226213 2024-03-19 01:01:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 01:01:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:42 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226213 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226213 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17226213 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-19 01:01:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-19 01:01:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 01:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:43 proc_lsst.multi:170 [INFO] [multi] job_id 17226214 proc_lsst.multi INFO: [multi] job_id 17226214 2024-03-19 01:01:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:43 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226214 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226214 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17226214 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-19 01:01:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-19 01:01:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:43 proc_lsst.multi:170 [INFO] [multi] job_id 17226215 proc_lsst.multi INFO: [multi] job_id 17226215 2024-03-19 01:01:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-19 01:01:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:43 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226215 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226215 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17226215 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-19 01:01:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-19 01:01:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-19 01:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:44 proc_lsst.multi:170 [INFO] [multi] job_id 17226216 proc_lsst.multi INFO: [multi] job_id 17226216 2024-03-19 01:01:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-19 01:01:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:44 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226216 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226216 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17226216 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-19 01:01:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-19 01:01:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-19 01:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:44 proc_lsst.multi:170 [INFO] [multi] job_id 17226217 proc_lsst.multi INFO: [multi] job_id 17226217 2024-03-19 01:01:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-19 01:01:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:44 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226217 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226217 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17226217 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-19 01:01:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-19 01:01:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-19 01:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:45 proc_lsst.multi:170 [INFO] [multi] job_id 17226218 proc_lsst.multi INFO: [multi] job_id 17226218 2024-03-19 01:01:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-19 01:01:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:45 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226218 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226218 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17226218 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-19 01:01:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-19 01:01:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-19 01:01:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:46 proc_lsst.multi:170 [INFO] [multi] job_id 17226219 proc_lsst.multi INFO: [multi] job_id 17226219 2024-03-19 01:01:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-19 01:01:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:46 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226219 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226219 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17226219 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-19 01:01:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-19 01:01:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-19 01:01:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a77f6b2f06718f128c6e5db92d77ec55 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 8be44b864314cd910ff46986e7bd7ad4 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/878905c1-cddc-457a-acc4-ba31697cb8d2_characterizeImage_1101466_14.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/878905c1-cddc-457a-acc4-ba31697cb8d2_characterizeImage_1101466_14.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/9fdd1b75-6f06-40eb-8634-e78107c94894_isr_1101466_14.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/9fdd1b75-6f06-40eb-8634-e78107c94894_isr_1101466_14.stderr 2024-03-19 01:01:47 proc_lsst.multi:170 [INFO] [multi] job_id 17226220 proc_lsst.multi INFO: [multi] job_id 17226220 2024-03-19 01:01:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-19 01:01:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:47 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226220 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226220 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17226220 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-19 01:01:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-19 01:01:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-19 01:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:47 proc_lsst.multi:170 [INFO] [multi] job_id 17226221 proc_lsst.multi INFO: [multi] job_id 17226221 2024-03-19 01:01:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-19 01:01:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:47 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226221 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226221 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17226221 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-19 01:01:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-19 01:01:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-19 01:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 060120feb0fe5bfb1281a78353404483 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash d4d34ff78331d51e2c7b8a8bae9c78c5 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162075800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-19 01:01:48 proc_lsst.multi:170 [INFO] [multi] job_id 17226222 proc_lsst.multi INFO: [multi] job_id 17226222 2024-03-19 01:01:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-19 01:01:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:48 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226222 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226222 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17226222 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-19 01:01:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-19 01:01:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-19 01:01:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/47839087-0c61-4013-bcc4-86caedbbae62_characterizeImage_1101475_52.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/47839087-0c61-4013-bcc4-86caedbbae62_characterizeImage_1101475_52.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/f21088fb-738d-4b7e-a8f6-aaa1b96253ef_isr_1101475_52.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/f21088fb-738d-4b7e-a8f6-aaa1b96253ef_isr_1101475_52.stderr parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:49 proc_lsst.multi:170 [INFO] [multi] job_id 17226223 proc_lsst.multi INFO: [multi] job_id 17226223 2024-03-19 01:01:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-19 01:01:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:49 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226223 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226223 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17226223 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-19 01:01:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-19 01:01:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-19 01:01:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f45c84fb6c23d9ab8a2815db96f0a8ea with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash e95e1598bc450e1e1cf2e56062aabb5a parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162077060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/eeae8028-2929-47c4-830b-a8554037e194_characterizeImage_1101475_25.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/eeae8028-2929-47c4-830b-a8554037e194_characterizeImage_1101475_25.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/883c156f-820d-4003-8a5d-b5005c329001_isr_1101475_25.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/883c156f-820d-4003-8a5d-b5005c329001_isr_1101475_25.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2c69614a67cc80b6644ffa9df26f8f3 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 813db91eab3fa515478e2b8dd34b0bc3 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162077380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/270bf27b-6e1f-4d52-8279-486140c6c68c_characterizeImage_1101469_5.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/270bf27b-6e1f-4d52-8279-486140c6c68c_characterizeImage_1101469_5.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/c93f2502-9df9-4411-8381-d72b51589f7d_isr_1101469_5.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/c93f2502-9df9-4411-8381-d72b51589f7d_isr_1101469_5.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 525772b6095d2670a5d9a2b033e39560 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash c24cb5fb4d80bce8a462e502ae0481af parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162098680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/74d61190-e67b-43bc-8c23-7b897d94851e_characterizeImage_1101472_24.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/74d61190-e67b-43bc-8c23-7b897d94851e_characterizeImage_1101472_24.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/20d45f7b-c1a7-49de-bd1f-1a7c0014215b_isr_1101472_24.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/20d45f7b-c1a7-49de-bd1f-1a7c0014215b_isr_1101472_24.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1dfcef87c8ebb51970d6b181368339f with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 7888dbafbe62a45e1dabf25698d5aff0 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162098360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/6d17a6a4-82d4-401f-8720-8b69d9b8aca7_characterizeImage_1101475_45.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/6d17a6a4-82d4-401f-8720-8b69d9b8aca7_characterizeImage_1101475_45.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/79352830-9bde-426a-bfda-833a12cf94e2_isr_1101475_45.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/79352830-9bde-426a-bfda-833a12cf94e2_isr_1101475_45.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 637cbdac4f0a68db099e4ba3014a6110 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash da98c6f723e75b6f313d8dd6b47fad4b parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162077e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/c75f1f98-2b9d-4b52-b8ee-62a17824ca1e_characterizeImage_1101472_31.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/c75f1f98-2b9d-4b52-b8ee-62a17824ca1e_characterizeImage_1101472_31.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/ebebec09-57bd-4e87-9672-7198b25ef3de_isr_1101472_31.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/ebebec09-57bd-4e87-9672-7198b25ef3de_isr_1101472_31.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 862f3443cee0aac780139fb14b7483d9 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash faa0d302f381facc97ed5e9099d100a0 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162077ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4405056c-6f80-4de3-aaeb-9aedc1bb05c8_characterizeImage_1101472_49.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4405056c-6f80-4de3-aaeb-9aedc1bb05c8_characterizeImage_1101472_49.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/3f709873-2353-4b28-a12c-1d988c30befb_isr_1101472_49.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/3f709873-2353-4b28-a12c-1d988c30befb_isr_1101472_49.stderr 2024-03-19 01:01:50 proc_lsst.multi:170 [INFO] [multi] job_id 17226224 proc_lsst.multi INFO: [multi] job_id 17226224 2024-03-19 01:01:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-19 01:01:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226224 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226224 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17226224 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-19 01:01:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-19 01:01:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-19 01:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:52 proc_lsst.multi:170 [INFO] [multi] job_id 17226225 proc_lsst.multi INFO: [multi] job_id 17226225 2024-03-19 01:01:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-19 01:01:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226225 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226225 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17226225 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-19 01:01:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-19 01:01:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-19 01:01:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 495fa0cf9d1f2cdf0110fda6a76f9a03 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash b9dde1540f513bc0feb4dd4ed33c1648 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162076660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/44f91728-b4fa-41dd-8108-f7b1c5d6c648_characterizeImage_1101469_50.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/44f91728-b4fa-41dd-8108-f7b1c5d6c648_characterizeImage_1101469_50.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/b03fceca-34bf-4c06-9f89-1815b147dfed_isr_1101469_50.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/b03fceca-34bf-4c06-9f89-1815b147dfed_isr_1101469_50.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbc555ae225415e6dd2099e168f17952 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash e5ff96be9856ecdf4c1f4603900c0378 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620989a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/06a9bae1-ec87-484c-8cda-4112c1d7440b_characterizeImage_1101472_18.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/06a9bae1-ec87-484c-8cda-4112c1d7440b_characterizeImage_1101472_18.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/4e522a7d-935f-4a81-83ce-6992e7f88c3f_isr_1101472_18.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/4e522a7d-935f-4a81-83ce-6992e7f88c3f_isr_1101472_18.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15b4d5bd0752fd4ee8cde14aae1cc2e5 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash bcbe5d910383353ec530f28573611689 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162076a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/d07bef3f-7f27-41e1-9b8e-fd1f9f4d04f8_characterizeImage_1101469_38.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/d07bef3f-7f27-41e1-9b8e-fd1f9f4d04f8_characterizeImage_1101469_38.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/b97df112-bd01-472a-b1aa-67ad92d8cec8_isr_1101469_38.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/b97df112-bd01-472a-b1aa-67ad92d8cec8_isr_1101469_38.stderr 2024-03-19 01:01:53 proc_lsst.multi:170 [INFO] [multi] job_id 17226226 proc_lsst.multi INFO: [multi] job_id 17226226 2024-03-19 01:01:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-19 01:01:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:53 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226226 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226226 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17226226 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-19 01:01:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-19 01:01:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-19 01:01:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8409c1dc68c70760db7d41925d54ad62 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash e21ccc325016de57f0c174e0b0b17436 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/fe470721-96d0-4871-beca-c0bdcb620ed3_characterizeImage_1101469_47.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/fe470721-96d0-4871-beca-c0bdcb620ed3_characterizeImage_1101469_47.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/d730b5da-2a9c-4966-8e9a-df581e647515_isr_1101469_47.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/d730b5da-2a9c-4966-8e9a-df581e647515_isr_1101469_47.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cb185c7427c0c399c667ad75fc67ca5 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 6bc292dc1d74538277dc000271ec7880 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162098fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/75d8076e-de60-4f7d-a4ee-670587b0935b_characterizeImage_1101475_35.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/75d8076e-de60-4f7d-a4ee-670587b0935b_characterizeImage_1101475_35.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/ee21e3fc-1cc3-49b6-927f-d719de56f27f_isr_1101475_35.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/ee21e3fc-1cc3-49b6-927f-d719de56f27f_isr_1101475_35.stderr 2024-03-19 01:01:54 proc_lsst.multi:170 [INFO] [multi] job_id 17226227 proc_lsst.multi INFO: [multi] job_id 17226227 2024-03-19 01:01:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-19 01:01:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:54 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226227 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226227 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17226227 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-19 01:01:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-19 01:01:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-19 01:01:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 01:01:55 proc_lsst.multi:170 [INFO] [multi] job_id 17226228 proc_lsst.multi INFO: [multi] job_id 17226228 2024-03-19 01:01:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-19 01:01:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:55 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226228 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226228 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17226228 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-19 01:01:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-19 01:01:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-19 01:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f3dd9714b56b32c594172f1de59c38e with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 280ef39981d3f4a66949846170e968cb parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162075f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/79ab60ee-11fa-4715-b284-b6a0d45f3baf_characterizeImage_1101472_15.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/79ab60ee-11fa-4715-b284-b6a0d45f3baf_characterizeImage_1101472_15.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d90f2de1-a08c-44b3-9c16-86ef039c9a53_isr_1101472_15.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d90f2de1-a08c-44b3-9c16-86ef039c9a53_isr_1101472_15.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 285e548c40a49b58487e177de55b865b with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 5f1e47458be3d847c4c1aaf30be56c38 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162075bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/ff1d0d4c-210c-476d-b738-00ef4c000a0a_characterizeImage_1101475_13.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/ff1d0d4c-210c-476d-b738-00ef4c000a0a_characterizeImage_1101475_13.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/edcb0cb2-cf49-4ac1-934a-7302c86d61da_isr_1101475_13.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/edcb0cb2-cf49-4ac1-934a-7302c86d61da_isr_1101475_13.stderr 2024-03-19 01:01:56 proc_lsst.multi:170 [INFO] [multi] job_id 17226229 proc_lsst.multi INFO: [multi] job_id 17226229 2024-03-19 01:01:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 01:01:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:56 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226229 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226229 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17226229 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-19 01:01:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-19 01:01:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 01:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-19 01:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 01:01:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c563fa4258154903994c1bfee26f1c9 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash f21f04f376ff51ec88340487e1019d29 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620779c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/580c5365-e0df-4099-a3d7-7f9ae4ef575b_characterizeImage_1101469_17.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/580c5365-e0df-4099-a3d7-7f9ae4ef575b_characterizeImage_1101469_17.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/6b54e84f-421c-4a64-98cf-34d2d7a474fa_isr_1101469_17.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/6b54e84f-421c-4a64-98cf-34d2d7a474fa_isr_1101469_17.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0229b327635e6ccac8b847078e5dc288 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 3fef8ef9b2a1894c1679941c169a69a3 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162098cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/4bf596b0-0d06-4a05-8e7b-93753a267111_characterizeImage_1101475_60.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/4bf596b0-0d06-4a05-8e7b-93753a267111_characterizeImage_1101475_60.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/f0695a6a-abb0-4e89-a2f6-5cbbfd4a03db_isr_1101475_60.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/f0695a6a-abb0-4e89-a2f6-5cbbfd4a03db_isr_1101475_60.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bee86952e7384edfa3611e4d524e307 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash aeb0ceee47fe18cbd564a4fc3350daf1 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620776a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/df2478d3-b391-4d93-a2cd-a87ad591e22a_characterizeImage_1101466_61.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/df2478d3-b391-4d93-a2cd-a87ad591e22a_characterizeImage_1101466_61.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/c5ad0679-1f9b-401a-b376-d7391f5eff31_isr_1101466_61.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/c5ad0679-1f9b-401a-b376-d7391f5eff31_isr_1101466_61.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62861f678f08201d6bcc9521d358610e with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 23c26ccd19a22f85d38cae24fcabc2af parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162076d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c5608d05-ccaf-41e1-b310-21842549b308_characterizeImage_1101466_60.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c5608d05-ccaf-41e1-b310-21842549b308_characterizeImage_1101466_60.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/e9f509f8-2e3b-4e74-8d95-b0a1fcf58467_isr_1101466_60.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/e9f509f8-2e3b-4e74-8d95-b0a1fcf58467_isr_1101466_60.stderr 2024-03-19 01:01:57 proc_lsst.multi:170 [INFO] [multi] job_id 17226230 proc_lsst.multi INFO: [multi] job_id 17226230 2024-03-19 01:01:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 01:01:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17226230 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17226230 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17226230 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-19 01:01:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-19 01:01:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 01:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 01:01:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3613e043ed4c424ab3471a81509b39a7 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 69c6ee7dbb7520b769d65c0365a65c3d parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162076340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/a5541e60-e8c8-46ed-a7cd-9e7539ba89a0_characterizeImage_1101472_36.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/a5541e60-e8c8-46ed-a7cd-9e7539ba89a0_characterizeImage_1101472_36.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a0c913fe-5618-4cf8-8c9e-c2d76bde504c_isr_1101472_36.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a0c913fe-5618-4cf8-8c9e-c2d76bde504c_isr_1101472_36.stderr 2024-03-19 01:01:58 proc_lsst.multi:170 [INFO] [multi] job_id 17226231 proc_lsst.multi INFO: [multi] job_id 17226231 2024-03-19 01:01:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 01:01:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 01:01:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17226231 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17226231 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17226231 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-19 01:01:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-19 01:01:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:01:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:01:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:01:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:01:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:01:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:01:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:01:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 01:01:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:01:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:01:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 01:01:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 01:01:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a94688c3ce86c1298b948daf736f410 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 05438464615ad439b02f8855b886eab6 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6947a0ef-1a11-4641-97e4-01254842c014_characterizeImage_1101469_11.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6947a0ef-1a11-4641-97e4-01254842c014_characterizeImage_1101469_11.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9bf29db4-dc51-4a74-842c-0bd76bccfcd8_isr_1101469_11.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9bf29db4-dc51-4a74-842c-0bd76bccfcd8_isr_1101469_11.stderr 2024-03-19 01:02:00 proc_lsst.multi:170 [INFO] [multi] job_id 17226232 proc_lsst.multi INFO: [multi] job_id 17226232 2024-03-19 01:02:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 01:02:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 01:02:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17226232 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17226232 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17226232 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-19 01:02:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-19 01:02:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:02:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:02:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:02:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:02:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:02:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:02:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:02:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:02:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:02:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:02:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 01:02:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:02:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:02:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 01:02:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 01:02:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 221e1dda3ae44d376261da4543dec132 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash e1dc3d31dc0dbc873244381fbea98a10 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/5b7627c3-83ca-4602-8749-6afe8f249e28_characterizeImage_1101475_5.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/5b7627c3-83ca-4602-8749-6afe8f249e28_characterizeImage_1101475_5.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c3b2650f-a76a-46ca-9e1c-47765660423a_isr_1101475_5.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c3b2650f-a76a-46ca-9e1c-47765660423a_isr_1101475_5.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac0967e0aa25cdeea605c2acd57c0954 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash b60cd0769a3db0dd224a7fea872d8bc1 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8e5ffebb-f698-426e-a0db-cbbb09680f62_characterizeImage_1101475_27.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8e5ffebb-f698-426e-a0db-cbbb09680f62_characterizeImage_1101475_27.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a787bad6-0143-401d-be21-6c0f842f5f8e_isr_1101475_27.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a787bad6-0143-401d-be21-6c0f842f5f8e_isr_1101475_27.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72cecce7eafeefa15de4f30207ed1021 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 054663d5027a8b0d1c9c61bd3d1804ee parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2387772d-c1fe-49d7-acf7-f237679602a3_characterizeImage_1101469_15.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2387772d-c1fe-49d7-acf7-f237679602a3_characterizeImage_1101469_15.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/fd3b038e-eb67-4d39-9c65-6626fac1aea6_isr_1101469_15.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/fd3b038e-eb67-4d39-9c65-6626fac1aea6_isr_1101469_15.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 261a687247cff155d551725d145f405a with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 4dabe638fe72d9416f5692fbd099a821 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/13fc318d-4289-449e-9140-ef1088f20a73_characterizeImage_1101475_24.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/13fc318d-4289-449e-9140-ef1088f20a73_characterizeImage_1101475_24.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/be20ba6e-dacc-488f-9408-143ee0d96bf9_isr_1101475_24.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/be20ba6e-dacc-488f-9408-143ee0d96bf9_isr_1101475_24.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa4abf574160bd3f825c451a5d1e5273 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash becf0c8764607f7664bfcc5807509053 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6da3c72f-a0b6-4efc-b775-9d656f1d8fd4_characterizeImage_1101469_14.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6da3c72f-a0b6-4efc-b775-9d656f1d8fd4_characterizeImage_1101469_14.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/f753dd15-fa99-4a80-b8d4-092080524cc2_isr_1101469_14.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/f753dd15-fa99-4a80-b8d4-092080524cc2_isr_1101469_14.stderr 2024-03-19 01:02:04 proc_lsst.multi:170 [INFO] [multi] job_id 17226233 proc_lsst.multi INFO: [multi] job_id 17226233 2024-03-19 01:02:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 01:02:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 01:02:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17226233 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17226233 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17226233 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-19 01:02:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-19 01:02:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:02:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:02:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:02:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 01:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:02:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 01:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 01:02:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9380fb67cdc2876fa8322fe3501f138a with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 1844612fa0f45868e57f3926fd0f6e09 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/03d23cc2-5b22-41ea-93f9-478e668a0c82_characterizeImage_1101466_34.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/03d23cc2-5b22-41ea-93f9-478e668a0c82_characterizeImage_1101466_34.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/2f037e97-5b49-4c82-8721-ca8c56ed0524_isr_1101466_34.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/2f037e97-5b49-4c82-8721-ca8c56ed0524_isr_1101466_34.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a17359d64882b506631fdba4c7d4fbe with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 65f8cab3a4db4f7b7afa738131e7f9dc parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/6621a70a-d337-4c77-aec1-33ea1c758628_characterizeImage_1101472_48.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/6621a70a-d337-4c77-aec1-33ea1c758628_characterizeImage_1101472_48.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/82d432a8-86d4-432a-b2ac-087a00cb475c_isr_1101472_48.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/82d432a8-86d4-432a-b2ac-087a00cb475c_isr_1101472_48.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ffa7f14ac9550dea15ccd10adf7b914 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash cb1fc970921beeca5240b115229f1285 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/83504901-9919-4c78-a8ef-2a7bad58d3a2_characterizeImage_1101469_32.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/83504901-9919-4c78-a8ef-2a7bad58d3a2_characterizeImage_1101469_32.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/6644c07c-681e-4821-971d-7b486b73f881_isr_1101469_32.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/6644c07c-681e-4821-971d-7b486b73f881_isr_1101469_32.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76d85694a8377991dc519cf238947e1a with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash f8a99589813d545632b2476be57afbab parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620beac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/fe77936e-b7d0-4172-b713-0bca6f0eb458_characterizeImage_1101469_40.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/fe77936e-b7d0-4172-b713-0bca6f0eb458_characterizeImage_1101469_40.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/fe6c6fd5-33fc-4b1a-a09b-cbf5d384e08e_isr_1101469_40.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/fe6c6fd5-33fc-4b1a-a09b-cbf5d384e08e_isr_1101469_40.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8613d9815377283956e49df3f5b9457c with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash ecff2e788a69b69ee8e788a6fa53b30e parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/8231f787-ff03-4b71-9dbc-52f8a7b5d375_characterizeImage_1101469_20.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/8231f787-ff03-4b71-9dbc-52f8a7b5d375_characterizeImage_1101469_20.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/902c7e5d-0f3c-413a-822c-f8914dd6afc5_isr_1101469_20.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/902c7e5d-0f3c-413a-822c-f8914dd6afc5_isr_1101469_20.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b8805cc94431968d258f70bac3ade5b with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 2fc5468478733b35412528a1c76389c0 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/6ce55a8b-1d2b-4d52-a7a0-707eadb6a4a2_characterizeImage_1101466_30.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/6ce55a8b-1d2b-4d52-a7a0-707eadb6a4a2_characterizeImage_1101466_30.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/d5bb8a7c-3abb-4a4b-865b-9a0765173e2a_isr_1101466_30.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/d5bb8a7c-3abb-4a4b-865b-9a0765173e2a_isr_1101466_30.stderr 2024-03-19 01:02:06 proc_lsst.multi:170 [INFO] [multi] job_id 17226237 proc_lsst.multi INFO: [multi] job_id 17226237 2024-03-19 01:02:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:02:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 01:02:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17226237 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17226237 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17226237 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-19 01:02:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-19 01:02:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 01:02:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 01:02:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 01:02:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:02:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 01:02:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 01:02:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:02:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:02:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 01:02:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:02:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 01:02:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 01:02:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 01:02:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 01:02:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 01:02:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,10.64.65.9,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54556 --result_port=54562 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78a9c8941791beafe72e00cd86f16819 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 427249a4c400f17c4726b35a76e8ea8d parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/fb9705a0-5310-483c-86cd-b9a986e383fc_characterizeImage_1101466_1.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/fb9705a0-5310-483c-86cd-b9a986e383fc_characterizeImage_1101466_1.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/85acd5cf-1227-4f17-989b-90ba8e95e0e9_isr_1101466_1.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/85acd5cf-1227-4f17-989b-90ba8e95e0e9_isr_1101466_1.stderr 2024-03-19 01:02:07 proc_lsst.multi:170 [INFO] [multi] job_id 17226238 proc_lsst.multi INFO: [multi] job_id 17226238 2024-03-19 01:02:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-19 01:02:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 01:02:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17226238 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17226238 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17226238 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:02:07 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:02:07 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cd88c9f2e953749d7e1468792cf94fb with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 16a74798baee91ff29d5b263b0011518 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/d4add97f-5875-46d5-a9ff-9440e608356c_characterizeImage_1101475_61.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/d4add97f-5875-46d5-a9ff-9440e608356c_characterizeImage_1101475_61.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/b4dac980-e2af-4e4c-a99f-72063e4db717_isr_1101475_61.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/b4dac980-e2af-4e4c-a99f-72063e4db717_isr_1101475_61.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 322eba8583b624a92993c63ed855cbfb with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 979c4f9f3a8f942fa94caa3ee195972e parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/5e5a1037-ea0a-4a57-a366-dd5b3a210805_characterizeImage_1101466_18.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/5e5a1037-ea0a-4a57-a366-dd5b3a210805_characterizeImage_1101466_18.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/7dfd929e-2b52-49e7-a176-457157262a80_isr_1101466_18.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/7dfd929e-2b52-49e7-a176-457157262a80_isr_1101466_18.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69dede63ddd1174f494c448925c0e240 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 7b4d65aa96fae38779d83417f6e49f18 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/106bdb6a-b607-42f1-a27e-2e26a82a558a_characterizeImage_1101466_17.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/106bdb6a-b607-42f1-a27e-2e26a82a558a_characterizeImage_1101466_17.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/abcdc947-c607-42b2-9074-2f0e23e0c432_isr_1101466_17.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/abcdc947-c607-42b2-9074-2f0e23e0c432_isr_1101466_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:08 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 825078ac95eac8929c65c5cb49fc574a with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 97b71c68b81118f348a9e6eb10f8262d parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:09 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e5efe47c-acec-4352-b499-eb5d44cd9087_characterizeImage_1101475_44.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e5efe47c-acec-4352-b499-eb5d44cd9087_characterizeImage_1101475_44.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a928af09-9daa-4a04-9d86-d3ad8a3a3191_isr_1101475_44.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a928af09-9daa-4a04-9d86-d3ad8a3a3191_isr_1101475_44.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e8ce8e4543973177b946f70076da8af with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 6a339c2703ba7e4a66e3291cb3a151be parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/969bf8b1-bf98-41b8-83d3-deb012d8f271_characterizeImage_1101469_61.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/969bf8b1-bf98-41b8-83d3-deb012d8f271_characterizeImage_1101469_61.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/910bad8c-b87c-4b6d-b0a6-14278b1d7d05_isr_1101469_61.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/910bad8c-b87c-4b6d-b0a6-14278b1d7d05_isr_1101469_61.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfbaeb8411ce1f1c31542e2d8109c4d7 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 29fadb05ff1f9ebc7c32001419101ef8 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/d83c6c4d-b95b-450b-85cf-c6b2f73058ba_characterizeImage_1101475_51.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/d83c6c4d-b95b-450b-85cf-c6b2f73058ba_characterizeImage_1101475_51.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c838db30-7d10-4a8b-ae11-9fbb79963bf6_isr_1101475_51.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c838db30-7d10-4a8b-ae11-9fbb79963bf6_isr_1101475_51.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fb3291da622319dc9e3aaba5c966572 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 982ba0fbc22265f27f0ae9d2fe0add30 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c695400b-ed2b-4bf3-9b12-687215a1c4bd_characterizeImage_1101475_16.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c695400b-ed2b-4bf3-9b12-687215a1c4bd_characterizeImage_1101475_16.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/cf06c4e0-cda1-4e18-96bf-cc861ffa2793_isr_1101475_16.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/cf06c4e0-cda1-4e18-96bf-cc861ffa2793_isr_1101475_16.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 592dfb18142f25d889404a2d34178b57 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 978d5e4ccc000685ed65d9fd101b4e22 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/21185a09-1896-4428-b512-4c0f20819bb6_characterizeImage_1101472_39.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/21185a09-1896-4428-b512-4c0f20819bb6_characterizeImage_1101472_39.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/f06826c7-8a02-4741-9614-b903b7946e49_isr_1101472_39.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/f06826c7-8a02-4741-9614-b903b7946e49_isr_1101472_39.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ade1f4f96edbcc63975e0cd51e23aa8 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 8b5cc6dd0c3297316f6268c56e31e6fb parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/16483b90-c8c3-4295-9489-bc2036c87424_characterizeImage_1101469_7.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/16483b90-c8c3-4295-9489-bc2036c87424_characterizeImage_1101469_7.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/4ec9eab0-0777-4d82-ac07-c8c910274468_isr_1101469_7.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/4ec9eab0-0777-4d82-ac07-c8c910274468_isr_1101469_7.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d71fbc7e8320ae673f7505d4cfed6092 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 089f8d87fae18de8d3a19ec8186c6c28 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/0049625a-9cb3-41f5-8796-52f6d6126fb7_characterizeImage_1101472_61.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/0049625a-9cb3-41f5-8796-52f6d6126fb7_characterizeImage_1101472_61.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/24afbb0f-fa54-4388-8917-718b8cb96d9d_isr_1101472_61.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/24afbb0f-fa54-4388-8917-718b8cb96d9d_isr_1101472_61.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eab71dd05f39676fa04d6099ff799d8c with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash ed6116c934d9c12eb43bfe40298f86e9 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/1c80089c-af42-4a1f-a023-b95b5b5d5422_characterizeImage_1101472_1.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/1c80089c-af42-4a1f-a023-b95b5b5d5422_characterizeImage_1101472_1.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/87b6f434-d5aa-481d-9fe5-f67b6a6368de_isr_1101472_1.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/87b6f434-d5aa-481d-9fe5-f67b6a6368de_isr_1101472_1.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86287a58e7df3a8b009334174aa68a97 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 19ef10963e6e7a9508c51bde2aae1925 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/8506dfee-d3a6-44c8-b2ec-998082e449ca_characterizeImage_1101469_57.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/8506dfee-d3a6-44c8-b2ec-998082e449ca_characterizeImage_1101469_57.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/92503d1f-0854-4384-8b87-8af4b5b29592_isr_1101469_57.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/92503d1f-0854-4384-8b87-8af4b5b29592_isr_1101469_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:11 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:11 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 645dddd92f2ffe59ecd087c41a144528 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash df5aeee5d0ab3abf9aa4948b5c34fce3 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/975abd01-2068-44d3-b9a1-1c1ea617f4af_characterizeImage_1101466_57.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/975abd01-2068-44d3-b9a1-1c1ea617f4af_characterizeImage_1101466_57.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/451646ff-f31c-4783-8358-09a53327cdfc_isr_1101466_57.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/451646ff-f31c-4783-8358-09a53327cdfc_isr_1101466_57.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63f1ead9a6a553725f428f0767de3bdb with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 79eb16f3a0ed319adb4b98f3d718c74d parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/18446002-9c1e-4765-9ed7-464bc2b481ff_characterizeImage_1101469_34.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/18446002-9c1e-4765-9ed7-464bc2b481ff_characterizeImage_1101469_34.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/1f0a726a-a272-4870-825a-74be5c8ca6a5_isr_1101469_34.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/1f0a726a-a272-4870-825a-74be5c8ca6a5_isr_1101469_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:12 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:13 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64ca7c9c857523ffa0356315f0c64300 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 46caecc579c564289e6955f45b766d0f parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/20232d0a-58ba-48fc-aa54-b533f4315330_characterizeImage_1101475_48.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/20232d0a-58ba-48fc-aa54-b533f4315330_characterizeImage_1101475_48.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/3831aa3c-916b-4598-82f8-a236a8139562_isr_1101475_48.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/3831aa3c-916b-4598-82f8-a236a8139562_isr_1101475_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:14 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b859da5ff4348deffcfd351229179b2e with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash be0b0f9be47063db9fa119c7431e7acd parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/ffcb7d03-1e1f-4398-9e27-4398ff459ae9_characterizeImage_1101475_22.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/ffcb7d03-1e1f-4398-9e27-4398ff459ae9_characterizeImage_1101475_22.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/8c296908-2e32-4cfb-bfbb-0ba0c6728262_isr_1101475_22.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/8c296908-2e32-4cfb-bfbb-0ba0c6728262_isr_1101475_22.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d23bad82b7dff13f184e7903a8f0a0e4 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 49a6c3458d5e4acdc4070c697518e8bd parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/7e96e7c6-f527-4608-a2bb-6e56b3ccf724_characterizeImage_1101472_41.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/7e96e7c6-f527-4608-a2bb-6e56b3ccf724_characterizeImage_1101472_41.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/47ff6b20-8235-4c5c-add3-48c0470d7654_isr_1101472_41.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/47ff6b20-8235-4c5c-add3-48c0470d7654_isr_1101472_41.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 057bfc91831296426f7640aefbca1827 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash d0b38e236aedc352123bfcc09b7627e1 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8757ec0b-e726-4075-82fe-d119568bea5d_characterizeImage_1101475_31.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8757ec0b-e726-4075-82fe-d119568bea5d_characterizeImage_1101475_31.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/bba5d219-d04e-4fd4-813a-ff12d5d43380_isr_1101475_31.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/bba5d219-d04e-4fd4-813a-ff12d5d43380_isr_1101475_31.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83fe9efe813a179c7b2f2354ae0716c5 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 33baa97308adde9ae27015e5231efd7d parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620be7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7c79a3c6-e56d-4b40-bd06-8eb914dec934_characterizeImage_1101466_2.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7c79a3c6-e56d-4b40-bd06-8eb914dec934_characterizeImage_1101466_2.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/65d2968d-c993-4b59-90dc-c05f9622a6b5_isr_1101466_2.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/65d2968d-c993-4b59-90dc-c05f9622a6b5_isr_1101466_2.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c1957abb99d6af37a718a66e780e3f with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 8d69ad1815f761cb00a2ed864f17513a parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/ae7bbbdd-dd85-416d-a437-236ed3689694_characterizeImage_1101475_50.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/ae7bbbdd-dd85-416d-a437-236ed3689694_characterizeImage_1101475_50.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/75373f35-b63f-42a3-88db-42e9d3fd9489_isr_1101475_50.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/75373f35-b63f-42a3-88db-42e9d3fd9489_isr_1101475_50.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe87653938016b710eb9d0682060e6c3 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 37bddcb0d637d76ab4698989300ccb49 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/dd1eccc9-e2cc-44ae-9e6a-98954b4d4fef_characterizeImage_1101475_14.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/dd1eccc9-e2cc-44ae-9e6a-98954b4d4fef_characterizeImage_1101475_14.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/9f39b649-1cba-4bfb-bc7f-cf967bf571a8_isr_1101475_14.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/9f39b649-1cba-4bfb-bc7f-cf967bf571a8_isr_1101475_14.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 856205f449dc0d040cc5765ed01ba897 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 4f756bffec60563ac1104b3f5b00160e parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bf740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/8c9199be-a8df-4956-b55c-79739cbc71a1_characterizeImage_1101466_27.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/8c9199be-a8df-4956-b55c-79739cbc71a1_characterizeImage_1101466_27.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/1627d05b-238f-4787-96fc-c084dfc5e577_isr_1101466_27.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/1627d05b-238f-4787-96fc-c084dfc5e577_isr_1101466_27.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc52b7c041b61b9d1d7099b65bceb702 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 480e035f32de70d8c3b684c18baf4d91 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d52259b2-abb8-4283-90d5-9004d68e8f70_characterizeImage_1101472_51.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d52259b2-abb8-4283-90d5-9004d68e8f70_characterizeImage_1101472_51.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/baa6773c-5bee-47ee-9200-b434aceed1c3_isr_1101472_51.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/baa6773c-5bee-47ee-9200-b434aceed1c3_isr_1101472_51.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25f0ce7decb222e21d66b7cb50629c2a with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash dbf969a86967f6f7914750b0b8a1b84d parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/5fec3049-e658-48be-b1cb-0a077d5afb4f_characterizeImage_1101469_12.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/5fec3049-e658-48be-b1cb-0a077d5afb4f_characterizeImage_1101469_12.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/1f8b4bca-f081-48bf-8500-d4dea2fa9b71_isr_1101469_12.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/1f8b4bca-f081-48bf-8500-d4dea2fa9b71_isr_1101469_12.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ed0b5bf13447d78a63ac57c7b6e8d14 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 7ded58828fc819f0ae4e01c6f896fc3b parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620be160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/612de6a8-b7cf-4cd9-a5b9-c2b77805e11a_characterizeImage_1101466_41.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/612de6a8-b7cf-4cd9-a5b9-c2b77805e11a_characterizeImage_1101466_41.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/0eca8887-346f-4a62-83da-cd9b5746ce7a_isr_1101466_41.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/0eca8887-346f-4a62-83da-cd9b5746ce7a_isr_1101466_41.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f7b226a61534258b5ff472c744377ee with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 3ce7307e1a91a892f371d0a4fbf9dc15 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/9a4201a7-894f-48b5-8251-556a75b7317b_characterizeImage_1101472_3.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/9a4201a7-894f-48b5-8251-556a75b7317b_characterizeImage_1101472_3.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1b14e5a4-b5fa-4d59-ac92-020988d48749_isr_1101472_3.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1b14e5a4-b5fa-4d59-ac92-020988d48749_isr_1101472_3.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d788c8d7e092b124b6edf2d0c142af23 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash af01352cffa68d0d9f4a5ed7135648bd parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/df28707d-4d0b-4a9e-8ea7-323fda868854_characterizeImage_1101475_29.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/df28707d-4d0b-4a9e-8ea7-323fda868854_characterizeImage_1101475_29.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/e1403547-6cf5-48bd-bc95-b8aa579d83e2_isr_1101475_29.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/e1403547-6cf5-48bd-bc95-b8aa579d83e2_isr_1101475_29.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0b7250037fa047856c2e7a4a4ad9e8f with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash eeb684ba0fb25c4f3a1146311cc81db4 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/4dc37ee3-8c99-4447-8de9-f145e33f34cf_characterizeImage_1101466_6.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/4dc37ee3-8c99-4447-8de9-f145e33f34cf_characterizeImage_1101466_6.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/e1486017-5913-4a92-80e2-c58842513d85_isr_1101466_6.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/e1486017-5913-4a92-80e2-c58842513d85_isr_1101466_6.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a0923a4491cb2587f24e0e7ec1e2412 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 9438f0911fb1f4e6fc2b47e274e8a358 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/ee7b1ab5-87eb-4397-82d7-1e3232b99f6e_characterizeImage_1101466_12.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/ee7b1ab5-87eb-4397-82d7-1e3232b99f6e_characterizeImage_1101466_12.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/8f9e8ad0-3fb4-4e50-92b5-517fac91e6e2_isr_1101466_12.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/8f9e8ad0-3fb4-4e50-92b5-517fac91e6e2_isr_1101466_12.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77117e7e7172b967e83e8bdf78abee05 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 33b3a381b5b144868b384fc45ee280b4 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d8a9d7e1-fa96-4fb2-8dab-79e731b9b976_characterizeImage_1101472_6.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d8a9d7e1-fa96-4fb2-8dab-79e731b9b976_characterizeImage_1101472_6.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/6b9d2ba9-b619-41ef-95c8-ebc93097cab2_isr_1101472_6.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/6b9d2ba9-b619-41ef-95c8-ebc93097cab2_isr_1101472_6.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 137bab43adee9cb802dec1e604d68510 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash b5b698a78fd4766083ef000553c3bb82 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/5421573e-2683-49f9-b486-587fb19b8e57_characterizeImage_1101466_16.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/5421573e-2683-49f9-b486-587fb19b8e57_characterizeImage_1101466_16.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/f1284538-f04e-4432-b289-3971fab52846_isr_1101466_16.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/f1284538-f04e-4432-b289-3971fab52846_isr_1101466_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:15 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e452e36e7c87aa8ec5c28cb60d4c073d with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 05fae351987ce0861f663c7cf71c6f8d parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/cf2318d1-684f-4227-8d9b-7d76a32a75e1_characterizeImage_1101475_54.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/cf2318d1-684f-4227-8d9b-7d76a32a75e1_characterizeImage_1101475_54.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/d2652ce8-aeeb-4bcc-8c2d-85b9a7869bcf_isr_1101475_54.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/d2652ce8-aeeb-4bcc-8c2d-85b9a7869bcf_isr_1101475_54.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b403dc2be1c0b69b63728b7e08c7bb1c with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash d3a1624636349fa458fa37656a62d449 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160add8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/76461483-7784-4577-9c5f-8a5113a61b7b_characterizeImage_1101475_28.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/76461483-7784-4577-9c5f-8a5113a61b7b_characterizeImage_1101475_28.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/3aa254e1-30ed-4598-b0ab-fe2c763885bb_isr_1101475_28.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/3aa254e1-30ed-4598-b0ab-fe2c763885bb_isr_1101475_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:16 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cb4f1ad9b97a56ddceb1fd553f4b527 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 893fae4dd497ae43997694199876e070 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/dca6b1fc-d5fc-443c-9b14-ef43b7d121c1_characterizeImage_1101469_60.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/dca6b1fc-d5fc-443c-9b14-ef43b7d121c1_characterizeImage_1101469_60.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/5ceba667-89a7-4d96-be98-164a8bd8848e_isr_1101469_60.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/5ceba667-89a7-4d96-be98-164a8bd8848e_isr_1101469_60.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b2cab659e6d9b78223856e5ea5fe2e6 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 4b2126dba4926f2ecd318c8f35429c9f parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620be480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/80e190d1-9cec-4438-b6e3-83a29316ab21_characterizeImage_1101469_37.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/80e190d1-9cec-4438-b6e3-83a29316ab21_characterizeImage_1101469_37.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9d5844cf-dff2-4f70-97d2-ba0df7a8e5e8_isr_1101469_37.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9d5844cf-dff2-4f70-97d2-ba0df7a8e5e8_isr_1101469_37.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e904cdb09cfd5460141ad2d557280eb2 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 479f17beec169e9ce2d1933bcbd7e3bb parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/058837e1-73bd-4b61-8346-d824e4454371_characterizeImage_1101475_53.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/058837e1-73bd-4b61-8346-d824e4454371_characterizeImage_1101475_53.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/0256731b-82c3-4768-a405-ef6d36100719_isr_1101475_53.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/0256731b-82c3-4768-a405-ef6d36100719_isr_1101475_53.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87c93e320ecdbeb5143ad63bbd869f17 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash e4da74f386cd389959428d49819118ce parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/92d7bbb6-900e-4f50-a39b-bd96beb90e3c_characterizeImage_1101472_58.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/92d7bbb6-900e-4f50-a39b-bd96beb90e3c_characterizeImage_1101472_58.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1e8c108d-03e1-4a1b-839d-88027c642972_isr_1101472_58.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1e8c108d-03e1-4a1b-839d-88027c642972_isr_1101472_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:17 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:19 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:20 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ea0105206a9f69d631ff1fe35f2010b with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 4c5cd686589527ca3ac25e04e8a22146 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c03015fc-6115-4583-8b5c-12702bb13f94_characterizeImage_1101469_33.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c03015fc-6115-4583-8b5c-12702bb13f94_characterizeImage_1101469_33.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/e33c16f2-1616-42b0-bb87-13ea4a2c14a9_isr_1101469_33.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/e33c16f2-1616-42b0-bb87-13ea4a2c14a9_isr_1101469_33.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07b717f5a67bc23f4bd1d572ef3d6e7a with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash d321b617955d259054355cd98724e370 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/f69798d2-343b-4823-8425-e0edaf34406e_characterizeImage_1101472_52.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/f69798d2-343b-4823-8425-e0edaf34406e_characterizeImage_1101472_52.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/85c64e87-924a-4364-921c-bb36c48ebf26_isr_1101472_52.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/85c64e87-924a-4364-921c-bb36c48ebf26_isr_1101472_52.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2dcba502073c637a18f7d5272ff4b57 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash ea93c988c2bd14222d2300bcf6dfb4d4 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160add260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/30890ee0-a024-44c7-83a1-55699578a46b_characterizeImage_1101472_43.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/30890ee0-a024-44c7-83a1-55699578a46b_characterizeImage_1101472_43.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/577469ee-3f05-42fa-a5da-f724b53212a1_isr_1101472_43.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/577469ee-3f05-42fa-a5da-f724b53212a1_isr_1101472_43.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4f07381ad204ac36f34f4d7c2347432 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 41578f39ade8c5259fb14462ebffe85c parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c889e4bc-5b76-4ec9-b925-6d539b2f8c5e_characterizeImage_1101469_51.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c889e4bc-5b76-4ec9-b925-6d539b2f8c5e_characterizeImage_1101469_51.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/a1ea8027-f43f-41ce-8575-1c8a230a63c2_isr_1101469_51.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/a1ea8027-f43f-41ce-8575-1c8a230a63c2_isr_1101469_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:21 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6c619b5115f5df702f4837be8424a9c with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash ef291b36a9fed9db10a71f90c0245ab6 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ade520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8def3e0e-f98e-40f5-aa5a-1d9ba6035f09_characterizeImage_1101475_9.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8def3e0e-f98e-40f5-aa5a-1d9ba6035f09_characterizeImage_1101475_9.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c4b2906b-b320-4542-9034-bfc83bc17072_isr_1101475_9.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c4b2906b-b320-4542-9034-bfc83bc17072_isr_1101475_9.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d54de62e72d1c7056fb172316ed4e1fe with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 27c25db796e8fcde96b4633bb280cfed parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/876d0552-45dc-4c09-84eb-60fed53ae104_characterizeImage_1101469_27.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/876d0552-45dc-4c09-84eb-60fed53ae104_characterizeImage_1101469_27.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/07bf8b1c-53b6-41ee-a5e9-48bf3fb47b0c_isr_1101469_27.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/07bf8b1c-53b6-41ee-a5e9-48bf3fb47b0c_isr_1101469_27.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7fa8b95b733ff36dfaea16770aa325d with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 9faacc789c1423c88558d08a4f950e18 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160addbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8652ad9b-12d4-45f8-a2e9-4495c11b7e41_characterizeImage_1101472_56.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8652ad9b-12d4-45f8-a2e9-4495c11b7e41_characterizeImage_1101472_56.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/ac276ecc-c8ed-4563-9932-01075f450750_isr_1101472_56.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/ac276ecc-c8ed-4563-9932-01075f450750_isr_1101472_56.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 172a7b9c2c50a936decd80b6b6da9db1 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash cab63387aceb0c60b45d5d2651ab0f41 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/5b87aab7-3c86-42a5-be9e-a6b0015fe9f0_characterizeImage_1101469_13.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/5b87aab7-3c86-42a5-be9e-a6b0015fe9f0_characterizeImage_1101469_13.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0c3d90a1-c094-4f6d-afa4-04563fecef50_isr_1101469_13.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0c3d90a1-c094-4f6d-afa4-04563fecef50_isr_1101469_13.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75f71a82ea747ab7e4ef42c21ba64cef with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 8b4700b9636b2e719df10a8118904e93 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e2fb4f39-4f4f-41dd-a3e3-5f8170318bee_characterizeImage_1101475_41.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e2fb4f39-4f4f-41dd-a3e3-5f8170318bee_characterizeImage_1101475_41.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/f3da9084-9a7f-43a5-b665-b3233d46c4e4_isr_1101475_41.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/f3da9084-9a7f-43a5-b665-b3233d46c4e4_isr_1101475_41.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e009b887f1d6fefc8518cd1d9a8b589a with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash de76548a185a998323eeb88233f3b5c1 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/b4593fbc-8d9e-4f25-99f6-e6ce9890435c_characterizeImage_1101469_8.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/b4593fbc-8d9e-4f25-99f6-e6ce9890435c_characterizeImage_1101469_8.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0854749c-4e80-4c69-8bf4-6d22532817ad_isr_1101469_8.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0854749c-4e80-4c69-8bf4-6d22532817ad_isr_1101469_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:22 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:23 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f70fb7f68989b173ecf221f5795a7d1 with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash ae30306732ca831676d7810e548ac6d2 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160add580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/10668301-2f00-446c-bf78-d0b40ec4ffe5_characterizeImage_1101472_4.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/10668301-2f00-446c-bf78-d0b40ec4ffe5_characterizeImage_1101472_4.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/bae8b176-f066-4c97-ab5b-130dc77b25a3_isr_1101472_4.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/bae8b176-f066-4c97-ab5b-130dc77b25a3_isr_1101472_4.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b68c5ce72a57a12b0b390dbe96470ca with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 592b987316c099da5597f7a01ed455b6 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adf4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/114708e7-153b-436b-9ea0-2427d618c687_characterizeImage_1101475_40.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/114708e7-153b-436b-9ea0-2427d618c687_characterizeImage_1101475_40.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/59276905-783a-4806-82aa-b2aa0ea86aa4_isr_1101475_40.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/59276905-783a-4806-82aa-b2aa0ea86aa4_isr_1101475_40.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13df7afbcd766e7abb7b226ce7076b87 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 878d404e090385d0c431bf277aae9d5d parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c3eebaaa-b67e-4939-a656-6762a3b95905_characterizeImage_1101475_12.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c3eebaaa-b67e-4939-a656-6762a3b95905_characterizeImage_1101475_12.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/83fa3b67-6079-4b07-9b38-0e51704b49da_isr_1101475_12.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/83fa3b67-6079-4b07-9b38-0e51704b49da_isr_1101475_12.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8187cc8182fe5a890d7fcdf2170dbec9 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 3ff3299a7b0b953900c7b81e277c210c parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/a33085e2-63c6-4d01-bc41-c39f6b65c7e2_characterizeImage_1101472_23.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/a33085e2-63c6-4d01-bc41-c39f6b65c7e2_characterizeImage_1101472_23.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/daa158f2-4f1d-48ec-b2c9-bf44d73db330_isr_1101472_23.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/daa158f2-4f1d-48ec-b2c9-bf44d73db330_isr_1101472_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:23 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd10e464b694d4b66675de2d2e9427b9 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash f3e8d2a718ad666d4d535c41315c88ad parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/fd117811-7962-4f97-a44e-76f4b7100ff3_characterizeImage_1101475_18.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/fd117811-7962-4f97-a44e-76f4b7100ff3_characterizeImage_1101475_18.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/29a8fd3b-dfdf-4d88-9e0e-dbeacd906ec8_isr_1101475_18.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/29a8fd3b-dfdf-4d88-9e0e-dbeacd906ec8_isr_1101475_18.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 965e7f18f4c65092b8d884562d86c033 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 62ff9d2c404a14d441b38fbd8c0a8dbb parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/872e7947-15dc-4a7c-a782-eeee4ce0848f_characterizeImage_1101466_25.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/872e7947-15dc-4a7c-a782-eeee4ce0848f_characterizeImage_1101466_25.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b852a8c1-7c07-47f2-9d9d-a6a8ee9e111e_isr_1101466_25.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b852a8c1-7c07-47f2-9d9d-a6a8ee9e111e_isr_1101466_25.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ec956f819543eda17848ef2858b1722 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 7a9ded743542a90f85946e3cb2f952c0 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/58eebfb7-c355-4ee4-9719-5fbceaf26821_characterizeImage_1101472_14.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/58eebfb7-c355-4ee4-9719-5fbceaf26821_characterizeImage_1101472_14.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/66148eca-ca32-4955-8b97-4d67177efe3a_isr_1101472_14.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/66148eca-ca32-4955-8b97-4d67177efe3a_isr_1101472_14.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d34f324a24d1f5a8f04230e79427e41a with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 400facc4c6ac790bd1f1e33326a7778d parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/16646fea-60a9-410c-9c59-a7a42e34a1ed_characterizeImage_1101475_1.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/16646fea-60a9-410c-9c59-a7a42e34a1ed_characterizeImage_1101475_1.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/ff5b76d0-4715-4405-b0b4-4f519495d874_isr_1101475_1.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/ff5b76d0-4715-4405-b0b4-4f519495d874_isr_1101475_1.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4d5cb767f8cc880be98710e60f934e6 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 5a9ba02dbdbe839ad94f2ebb8674c5a0 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ade200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/70b46cbf-eeed-49f2-9896-8f93c18471c4_characterizeImage_1101466_49.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/70b46cbf-eeed-49f2-9896-8f93c18471c4_characterizeImage_1101466_49.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/65b55b3d-e736-47b5-8c29-2197ebe5fcd9_isr_1101466_49.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/65b55b3d-e736-47b5-8c29-2197ebe5fcd9_isr_1101466_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:24 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a744435acd9efc655e6366326385f71 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash a1006f3bae99f8a18f36aea71eec91ac parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b04e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d9a5ca31-6c67-481c-92d0-9db7743b5e77_characterizeImage_1101472_28.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d9a5ca31-6c67-481c-92d0-9db7743b5e77_characterizeImage_1101472_28.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0fb5ea05-6e6a-425e-9f72-5eb0d7827856_isr_1101472_28.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0fb5ea05-6e6a-425e-9f72-5eb0d7827856_isr_1101472_28.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b3ce8ca43958b354fc2eaab163226f2 with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 3e94c9f83d3aa9165bbc628418df93fd parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ade840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/e218ca9f-df2e-40bd-a88e-fecca922b35c_characterizeImage_1101469_62.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/e218ca9f-df2e-40bd-a88e-fecca922b35c_characterizeImage_1101469_62.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9328e366-e0ba-4ef8-aff6-0720c4d04ad4_isr_1101469_62.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9328e366-e0ba-4ef8-aff6-0720c4d04ad4_isr_1101469_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:25 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8acc4ba14562cd01858f27684dc21fb9 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash dfbe47c827048406d1ee430cdc2226b9 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/b5763be8-14fc-4be2-a503-d50f71d4029e_characterizeImage_1101469_3.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/b5763be8-14fc-4be2-a503-d50f71d4029e_characterizeImage_1101469_3.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/04bef5aa-f921-4bd6-8144-dcae6dc02c37_isr_1101469_3.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/04bef5aa-f921-4bd6-8144-dcae6dc02c37_isr_1101469_3.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28d31e901f1c0fade5c557ac56a0fdd4 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash fc6bf7ff492f7f7155346a240be31deb parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/ffcd6f5b-c496-4faf-b955-3a1b326fb974_characterizeImage_1101469_30.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/ffcd6f5b-c496-4faf-b955-3a1b326fb974_characterizeImage_1101469_30.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/71f1f39d-8384-4c3a-b287-4cf780910166_isr_1101469_30.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/71f1f39d-8384-4c3a-b287-4cf780910166_isr_1101469_30.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bff7d1bfdb01b3baa84fa23044121e94 with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash ff0c04f45cf2d2704887381b773a02b2 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b044a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:25 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2500a4d4-f34a-47e2-a1f8-8543d55c832d_characterizeImage_1101469_2.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2500a4d4-f34a-47e2-a1f8-8543d55c832d_characterizeImage_1101469_2.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0759c5ad-d155-4c25-8ded-27a17abcda06_isr_1101469_2.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0759c5ad-d155-4c25-8ded-27a17abcda06_isr_1101469_2.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0c1b6a26b6b912e0b9c9617a7b6c220 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 68dabeaac15037f5f034e6d199d36eae parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160addee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e92c796f-d527-4387-bd56-bd22d013d651_characterizeImage_1101466_37.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e92c796f-d527-4387-bd56-bd22d013d651_characterizeImage_1101466_37.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/5797468c-3f54-4a22-8a76-cb815eb74973_isr_1101466_37.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/5797468c-3f54-4a22-8a76-cb815eb74973_isr_1101466_37.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0130825765f1a67eaf7422a9cb0bf861 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 038c8b3b337271993289a025a2ee923f parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/47fb1d5d-5d06-47eb-9772-7d3c09839765_characterizeImage_1101469_48.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/47fb1d5d-5d06-47eb-9772-7d3c09839765_characterizeImage_1101469_48.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/174953cb-73af-4d17-9c86-889af3377824_isr_1101469_48.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/174953cb-73af-4d17-9c86-889af3377824_isr_1101469_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:26 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8895304d2070f19849e3beda0b5840f7 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash e6a2ab8852f72b2c7d47368f856f3d63 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/684b97be-e935-46f3-ab3f-127deae9e818_characterizeImage_1101466_53.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/684b97be-e935-46f3-ab3f-127deae9e818_characterizeImage_1101466_53.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/60152752-4214-4230-ac04-ceddfde70f54_isr_1101466_53.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/60152752-4214-4230-ac04-ceddfde70f54_isr_1101466_53.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a50873897920141a934af6e590844154 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 0db2c6cc9e7452e512ff83a7c286d0ed parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b047c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/22edbfb2-39d7-40a8-8b7a-100370e41ed2_characterizeImage_1101466_51.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/22edbfb2-39d7-40a8-8b7a-100370e41ed2_characterizeImage_1101466_51.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/6d7f8d49-6c35-4024-815f-cd14a75161ff_isr_1101466_51.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/6d7f8d49-6c35-4024-815f-cd14a75161ff_isr_1101466_51.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cebcb76ab8548e3818c1689fd0de30c0 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash e57e9b17b15213888c0a380fe6413fc7 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c0b0043b-186a-445c-a3aa-caed089ebdb3_characterizeImage_1101475_58.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c0b0043b-186a-445c-a3aa-caed089ebdb3_characterizeImage_1101475_58.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/91de3da8-a06c-4aeb-9ad2-9959faea33d0_isr_1101475_58.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/91de3da8-a06c-4aeb-9ad2-9959faea33d0_isr_1101475_58.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37a34ed0a45758e63d701aaa773d2a5b with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 508ef34dd0a72cfadb490f7347499c29 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:27 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/8227cfc3-2b46-4448-aad0-9b56a7d4a56d_characterizeImage_1101469_19.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/8227cfc3-2b46-4448-aad0-9b56a7d4a56d_characterizeImage_1101469_19.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9e3ff556-e6b0-46bf-a7a9-0e7dc587f4f8_isr_1101469_19.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9e3ff556-e6b0-46bf-a7a9-0e7dc587f4f8_isr_1101469_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:27 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:27 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:27 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ef6868147653757c73d8f315bc98a39 with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 5f607d9fb656f777f4cf97ef272edae5 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b04ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/cae0bcc7-3725-46df-971b-4878269e0177_characterizeImage_1101472_11.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/cae0bcc7-3725-46df-971b-4878269e0177_characterizeImage_1101472_11.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/f0ad033b-5650-4a4a-9616-aac1a904d8b1_isr_1101472_11.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/f0ad033b-5650-4a4a-9616-aac1a904d8b1_isr_1101472_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:28 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:28 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51efae0d916c381240cad3739267f880 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 0bf8f9e5529477c51e4c4cbd46a23bdc parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/db963836-3798-4422-9161-29e8be316af9_characterizeImage_1101475_17.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/db963836-3798-4422-9161-29e8be316af9_characterizeImage_1101475_17.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/b5bbf232-1902-4d62-a9c4-321531cab7db_isr_1101475_17.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/b5bbf232-1902-4d62-a9c4-321531cab7db_isr_1101475_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:28 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:28 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:29 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:29 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b95dd54e612c039dc06afa7cb33d683 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 784236897a757a4b7d2cb4214ddebc17 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/9f913d4a-5cbc-4b01-b9a6-8277764eac4f_characterizeImage_1101475_36.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/9f913d4a-5cbc-4b01-b9a6-8277764eac4f_characterizeImage_1101475_36.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/fc973936-98f3-40b2-ba0c-edddd5dd758f_isr_1101475_36.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/fc973936-98f3-40b2-ba0c-edddd5dd758f_isr_1101475_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:29 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:29 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:29 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dad88048df0e77c585d0567c7db8446e with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 35c3284ecb5e5d704c30209c623f1328 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/9c72b743-202d-4360-92a7-c81838929db2_characterizeImage_1101472_12.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/9c72b743-202d-4360-92a7-c81838929db2_characterizeImage_1101472_12.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/5825e854-9428-40eb-bc9e-230f73714cc0_isr_1101472_12.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/5825e854-9428-40eb-bc9e-230f73714cc0_isr_1101472_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:30 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00ad592d28f4b4d2639bf76fab03cddb with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 032898f2ac2281f61f4567d8a3f4902b parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/46d5592a-c9fa-427f-9897-221acafea56a_characterizeImage_1101472_19.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/46d5592a-c9fa-427f-9897-221acafea56a_characterizeImage_1101472_19.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/83c5883d-c370-4019-97d9-66c32afa7349_isr_1101472_19.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/83c5883d-c370-4019-97d9-66c32afa7349_isr_1101472_19.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5625ac272714e4627daec94bc3cce3e with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash bd5aa1929b87643823370ef1efc99c2a parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b07ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/7c0d3bd9-261a-4a4a-89f3-1b325914e223_characterizeImage_1101472_33.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/7c0d3bd9-261a-4a4a-89f3-1b325914e223_characterizeImage_1101472_33.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/94a64feb-25ec-47da-b524-eef58b7274c5_isr_1101472_33.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/94a64feb-25ec-47da-b524-eef58b7274c5_isr_1101472_33.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f35695329f493f2912474ab8330cf319 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 711066079e7519628a67da3d67129c6e parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c9ed3001-7e27-43a3-be41-5ba859b848c3_characterizeImage_1101466_19.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c9ed3001-7e27-43a3-be41-5ba859b848c3_characterizeImage_1101466_19.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/4644947b-288c-4dc6-bbe9-84ae3e8e15ed_isr_1101466_19.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/4644947b-288c-4dc6-bbe9-84ae3e8e15ed_isr_1101466_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:30 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:30 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cc7bb08c0751e2c8f1ab90e5d0f31ca with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash 6ea8d15001ce70ef6a92e4049a561e25 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b39620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e1d387f6-d655-4e30-86b2-4b813d8835ae_characterizeImage_1101475_21.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e1d387f6-d655-4e30-86b2-4b813d8835ae_characterizeImage_1101475_21.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/34f86c3b-3e29-419c-9dbd-b07e4d202f1a_isr_1101475_21.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/34f86c3b-3e29-419c-9dbd-b07e4d202f1a_isr_1101475_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:30 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ae4062f19fcc4a6ed25fa0f5c50d9dc with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 02e9622835aa3eb7d0678a07d731885c parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/318d59b9-3e5b-4ff0-8990-5e60db888b0a_characterizeImage_1101466_3.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/318d59b9-3e5b-4ff0-8990-5e60db888b0a_characterizeImage_1101466_3.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/fad05104-f53a-4749-a3c2-e169b74c18d4_isr_1101466_3.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/fad05104-f53a-4749-a3c2-e169b74c18d4_isr_1101466_3.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71d35fb29a46939e37c662edafb3e7ca with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash cb7f7373854be1ff8f4e42d424c0f090 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b39300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/5ccfa9cd-09ae-4ccb-b25c-95e8ae68f975_characterizeImage_1101475_49.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/5ccfa9cd-09ae-4ccb-b25c-95e8ae68f975_characterizeImage_1101475_49.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/e131cf5c-effc-43d4-9f28-96482a528d28_isr_1101475_49.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/e131cf5c-effc-43d4-9f28-96482a528d28_isr_1101475_49.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee351e1aaa2246b8257db97763138c3c with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash c6041e50821597dc7301039ca09fa50d parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/750275a7-c013-4afa-8695-80fa3364cb64_characterizeImage_1101466_21.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/750275a7-c013-4afa-8695-80fa3364cb64_characterizeImage_1101466_21.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/0ba448ae-f915-4146-902f-790bf9bcb204_isr_1101466_21.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/0ba448ae-f915-4146-902f-790bf9bcb204_isr_1101466_21.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f707811dc812c7a190b35b93b9c0a7be with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 1b4734bac2cbc7ae18c5df2493de23ff parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b076a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8d08afb4-7984-4a76-a1a2-fa879105761c_characterizeImage_1101472_13.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8d08afb4-7984-4a76-a1a2-fa879105761c_characterizeImage_1101472_13.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0e7d7b54-1118-47f5-9355-32cb3a24cc0c_isr_1101472_13.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0e7d7b54-1118-47f5-9355-32cb3a24cc0c_isr_1101472_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:31 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:31 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46b606fdce5a1bc41ec691d0eff22697 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash e601194eaf44f60771df1ba2b313ac97 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6f26a06a-0ae4-4dbf-869a-87e29412e118_characterizeImage_1101469_43.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6f26a06a-0ae4-4dbf-869a-87e29412e118_characterizeImage_1101469_43.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/e3709366-d659-498d-8d2a-8c53c10be719_isr_1101469_43.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/e3709366-d659-498d-8d2a-8c53c10be719_isr_1101469_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:31 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:31 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c709fbf4c3f3bb8f5c4bc0c2554b9ddf with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 4ffb22e6381a5c69d8c69ba067d10837 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b389a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/49e04028-2539-4317-a800-2ab88181d789_characterizeImage_1101475_57.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/49e04028-2539-4317-a800-2ab88181d789_characterizeImage_1101475_57.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/42fcf718-45b1-432d-b9bf-9d87a9299a71_isr_1101475_57.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/42fcf718-45b1-432d-b9bf-9d87a9299a71_isr_1101475_57.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22846a6b25b421f462a9a88572556a92 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 01c0785b78d4b08aa92bd61b942f2d4e parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/73e9ad96-afed-4f1c-843f-902882e77556_characterizeImage_1101469_45.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/73e9ad96-afed-4f1c-843f-902882e77556_characterizeImage_1101469_45.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0e7b2c7a-b73e-4cbc-8fde-7bcdc34bfe31_isr_1101469_45.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0e7b2c7a-b73e-4cbc-8fde-7bcdc34bfe31_isr_1101469_45.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ceadbdbfeb509c4482b289c8c7e843b with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash c5ec35425a7c6c4fc8659628016f2e4d parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b38fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/9022c339-1bbc-435a-98b4-a0c60c350907_characterizeImage_1101475_38.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/9022c339-1bbc-435a-98b4-a0c60c350907_characterizeImage_1101475_38.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/7a9452f6-e1a8-4c17-90fb-c1520a6a9321_isr_1101475_38.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/7a9452f6-e1a8-4c17-90fb-c1520a6a9321_isr_1101475_38.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1064999a9c6dc783e5f927e5b802661 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash e3b9be754f66f005c748407808f30eb8 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b38cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/cfa2a5be-826d-4085-a33b-f25bb0012d32_characterizeImage_1101472_42.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/cfa2a5be-826d-4085-a33b-f25bb0012d32_characterizeImage_1101472_42.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0ac8b262-d06c-43c8-b6c7-c6a3177105b2_isr_1101472_42.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/0ac8b262-d06c-43c8-b6c7-c6a3177105b2_isr_1101472_42.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9e191e662d8e5997b7d727a4c29a4cf with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 6c30641b85a49a07ef41bd35e5695c2d parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/770bcdd2-b4b4-4047-97a3-e7d3ba005835_characterizeImage_1101472_54.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/770bcdd2-b4b4-4047-97a3-e7d3ba005835_characterizeImage_1101472_54.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/bce8bcd2-8825-4208-b6a4-2427c7008bb8_isr_1101472_54.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/bce8bcd2-8825-4208-b6a4-2427c7008bb8_isr_1101472_54.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f53d1db92b2129d87da89574095cc80 with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash bcffdc4cff35794ff73d7f7e19c7e452 parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b39da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/155479cd-fc32-44c0-b7a7-591478690722_characterizeImage_1101469_23.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/155479cd-fc32-44c0-b7a7-591478690722_characterizeImage_1101469_23.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0cf69265-f21a-47f3-9d2a-d71354aab47e_isr_1101469_23.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/0cf69265-f21a-47f3-9d2a-d71354aab47e_isr_1101469_23.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72c26d280ee954c796da58d8c67fcc37 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 9f2d9a06e4ff7957b6b18210c114ed74 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/1e71bb91-a346-4fef-9912-854f6d9c5584_characterizeImage_1101466_46.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/1e71bb91-a346-4fef-9912-854f6d9c5584_characterizeImage_1101466_46.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/61902384-c23d-421e-afa9-bab7ed828b1a_isr_1101466_46.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/61902384-c23d-421e-afa9-bab7ed828b1a_isr_1101466_46.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e320154a72a2826c62a9b57134713b28 with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 1e669fa8ed16a6f437b417a4df9c4d20 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/6f51b9d1-de6b-4e05-b0d5-b68de7744272_characterizeImage_1101475_47.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/6f51b9d1-de6b-4e05-b0d5-b68de7744272_characterizeImage_1101475_47.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/65091aaa-5810-4d01-8d40-9b47b1a3c3fd_isr_1101475_47.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/65091aaa-5810-4d01-8d40-9b47b1a3c3fd_isr_1101475_47.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e83c53e1623642ff9314ff429b0dc16 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash f18b6a8d7ec2d07b650f128d6663fcc4 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b38360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8e25aee7-724d-42be-bccc-01e74dc3c3d6_characterizeImage_1101472_62.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8e25aee7-724d-42be-bccc-01e74dc3c3d6_characterizeImage_1101472_62.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d8c44fc5-fc95-4176-9edd-ad10052c17ef_isr_1101472_62.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d8c44fc5-fc95-4176-9edd-ad10052c17ef_isr_1101472_62.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7be9a659483b5a7dd61c9416b017b35 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 5187dfd745f73877afb70d5f6afc0743 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b38680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/4ffbbd51-899a-4647-84b2-cbdb640644c5_characterizeImage_1101469_9.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/4ffbbd51-899a-4647-84b2-cbdb640644c5_characterizeImage_1101469_9.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/84b95863-06a1-4cf2-8c6e-6b7ae453fc2b_isr_1101469_9.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/84b95863-06a1-4cf2-8c6e-6b7ae453fc2b_isr_1101469_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:32 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:32 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eac311c294742d64ce9935411eb8d0a1 with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash 98d401949c96ea30488a352d447769a5 parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/4319ca08-584b-4406-bee4-69cae37a5bb4_characterizeImage_1101466_52.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/4319ca08-584b-4406-bee4-69cae37a5bb4_characterizeImage_1101466_52.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b42e2bff-5c2d-498b-9978-2566a326afaf_isr_1101466_52.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b42e2bff-5c2d-498b-9978-2566a326afaf_isr_1101466_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:33 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bddd70368fd6e4efdc9d27c0bdb4a7d with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash bc20271746db8bb97c12ae6950267080 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/98926d4c-aa1c-4e51-8e00-69ec1a43a4fa_characterizeImage_1101466_35.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/98926d4c-aa1c-4e51-8e00-69ec1a43a4fa_characterizeImage_1101466_35.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/eb4c1013-34d9-46d1-a287-170f7ad61565_isr_1101466_35.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/eb4c1013-34d9-46d1-a287-170f7ad61565_isr_1101466_35.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54e18926c48fb556d739e4e773054f20 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash f21e48f5535963a804ccb7dcd4a17544 parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b39940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/53b54df1-1b84-4b6b-a2f6-11300bc8d41d_characterizeImage_1101466_42.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/53b54df1-1b84-4b6b-a2f6-11300bc8d41d_characterizeImage_1101466_42.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/13fa7b39-4750-4a77-8250-66fb28c14055_isr_1101466_42.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/13fa7b39-4750-4a77-8250-66fb28c14055_isr_1101466_42.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a71df94fc9ddd0fe6742739f51babcfe with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash c2f758a25c51a9ef274fe2eca56c18c3 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b07e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6193ace2-1c64-4586-90b9-b8a47c9ca6be_characterizeImage_1101469_58.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6193ace2-1c64-4586-90b9-b8a47c9ca6be_characterizeImage_1101469_58.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/c0868192-8826-4c81-8766-1539d547ff8f_isr_1101469_58.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/c0868192-8826-4c81-8766-1539d547ff8f_isr_1101469_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:35 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:37 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92dfc163d09179c0ba19cd2752d63452 with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 44d29916b610b0be85f499cd47a4a8f5 parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a3e7e109-3cca-4625-b9ef-cac36a74e2d5_characterizeImage_1101475_39.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a3e7e109-3cca-4625-b9ef-cac36a74e2d5_characterizeImage_1101475_39.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/eb32b3a5-cf01-410f-b60a-00b2e7a00c83_isr_1101475_39.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/eb32b3a5-cf01-410f-b60a-00b2e7a00c83_isr_1101475_39.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 080f3d8b4e2797e20989b4128e9c79f6 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash a206fcaf1af9b655878ffe5c665c4fa8 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b07380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/0a1c9f0f-028a-49c8-86cd-c597fd38305a_characterizeImage_1101469_25.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/0a1c9f0f-028a-49c8-86cd-c597fd38305a_characterizeImage_1101469_25.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/dcee3b48-3d24-4e69-9a29-e4be9c4c082d_isr_1101469_25.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/dcee3b48-3d24-4e69-9a29-e4be9c4c082d_isr_1101469_25.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a91de09c0d3189103c35e990fb510206 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 4fd242e9d92d4ed6379b1415edeb4a5b parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b063e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/972ad8ef-de97-4fef-b14d-7cde4d715215_characterizeImage_1101469_18.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/972ad8ef-de97-4fef-b14d-7cde4d715215_characterizeImage_1101469_18.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/98cb4190-3ef7-42d7-9a36-8d32a5310c92_isr_1101469_18.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/98cb4190-3ef7-42d7-9a36-8d32a5310c92_isr_1101469_18.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18aa59901520c799913a013f2729d603 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 76a3446e151d475c84412a75f3b75553 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b06700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/cf974039-f615-48e6-85e5-4d235a146327_characterizeImage_1101475_6.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/cf974039-f615-48e6-85e5-4d235a146327_characterizeImage_1101475_6.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/362cfe0b-b65f-458b-b263-ed70df027f1b_isr_1101475_6.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/362cfe0b-b65f-458b-b263-ed70df027f1b_isr_1101475_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:40 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31a02de03caa6c4fbb02c3b73eee134e with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 46be3a42139acecb7f8b5abb5d4d3cf5 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/d577b439-7712-4652-9ff0-4d8e230f6ccb_characterizeImage_1101475_26.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/d577b439-7712-4652-9ff0-4d8e230f6ccb_characterizeImage_1101475_26.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/64ed4f09-8598-4126-963d-c5ac991c7a1c_isr_1101475_26.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/64ed4f09-8598-4126-963d-c5ac991c7a1c_isr_1101475_26.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a31f7eced81abe7f4fd5d321af13ec87 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 4b5e72f670e37b1aebe3adcf48bddec9 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b06d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/3f9311d9-8b19-4bfc-ab6b-fb3bd92b9172_characterizeImage_1101469_46.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/3f9311d9-8b19-4bfc-ab6b-fb3bd92b9172_characterizeImage_1101469_46.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/d81e4646-5f52-438c-b141-b0cb35b8bef4_isr_1101469_46.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/d81e4646-5f52-438c-b141-b0cb35b8bef4_isr_1101469_46.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77dd0af6e235a28b92ae2b1755366b95 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash c8b2b274a948b8ea742cea2ed3bca543 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b07060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4014b455-e6ca-4426-b428-73df35fe99b8_characterizeImage_1101472_17.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4014b455-e6ca-4426-b428-73df35fe99b8_characterizeImage_1101472_17.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/6bf99484-a427-4fc6-8242-4c6c39037549_isr_1101472_17.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/6bf99484-a427-4fc6-8242-4c6c39037549_isr_1101472_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:41 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49c30c6c7703720eb4717d3b4358f3a0 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 1346181c18d7c64285cdb7a21c505be9 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b060c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/bf0c1ec6-e75e-4737-b59f-a824a49589dc_characterizeImage_1101472_16.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/bf0c1ec6-e75e-4737-b59f-a824a49589dc_characterizeImage_1101472_16.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d5b5d17f-f8fd-48e3-8ca3-697d0bf05c1a_isr_1101472_16.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d5b5d17f-f8fd-48e3-8ca3-697d0bf05c1a_isr_1101472_16.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 626a4471b4889a8c2d80209093a2ccd7 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash ac36ab023585f76f3bda4f0f9fea6d24 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a32dfb08-cedc-4588-aac6-ed45b0c54557_characterizeImage_1101475_19.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a32dfb08-cedc-4588-aac6-ed45b0c54557_characterizeImage_1101475_19.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/01624b05-ad30-475b-aa5e-49828a90c265_isr_1101475_19.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/01624b05-ad30-475b-aa5e-49828a90c265_isr_1101475_19.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 709e11be58faf3a37039a4b8c7f3e0a8 with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 77b012388d3e1bae2ab72b37985a85a1 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/075b50ef-9076-43c9-a6f7-f56866eb0314_characterizeImage_1101475_2.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/075b50ef-9076-43c9-a6f7-f56866eb0314_characterizeImage_1101475_2.stderr parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/5c74439b-3bfc-4428-bf6b-68ee62d68c2c_isr_1101475_2.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/5c74439b-3bfc-4428-bf6b-68ee62d68c2c_isr_1101475_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:42 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dcf33eb27747675d5d7f1dba73aecf3 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 2f61aec43960ae172cf7d0b2a6cd5ebf parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b06a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e459e5e0-4083-463e-a854-f9a1d97a7a8b_characterizeImage_1101466_50.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e459e5e0-4083-463e-a854-f9a1d97a7a8b_characterizeImage_1101466_50.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/8f566d6f-eceb-485f-9ec9-0296ce48286e_isr_1101466_50.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/8f566d6f-eceb-485f-9ec9-0296ce48286e_isr_1101466_50.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35530129eaf5753ffc3c1b63272e2355 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 99d88ca520c345227fa454c5498ce47c parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/b61b7a75-af1a-4de9-aacc-ddf934a7364e_characterizeImage_1101472_27.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/b61b7a75-af1a-4de9-aacc-ddf934a7364e_characterizeImage_1101472_27.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/6e0859f2-56a7-492d-9035-453410656bf2_isr_1101472_27.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/6e0859f2-56a7-492d-9035-453410656bf2_isr_1101472_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:43 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fe3bf02608f2e0335464562407c1b06 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 93500ea1284c69cb4b238c4e3cce6257 parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8ffc3177-1cca-414d-9f83-72d28f4791ef_characterizeImage_1101475_32.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8ffc3177-1cca-414d-9f83-72d28f4791ef_characterizeImage_1101475_32.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a765c1cb-9c4e-4b13-9864-63daaddc3502_isr_1101475_32.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a765c1cb-9c4e-4b13-9864-63daaddc3502_isr_1101475_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:45 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e852fd1a42282263c1b9f940570bf56 with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash 58e021e54ef6b98c0349b3357bc2840b parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c891c68f-13e2-4951-9fac-ba72794009e5_characterizeImage_1101475_8.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c891c68f-13e2-4951-9fac-ba72794009e5_characterizeImage_1101475_8.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a51960a8-9f54-4f2b-83ce-5bdbd1c1f418_isr_1101475_8.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/a51960a8-9f54-4f2b-83ce-5bdbd1c1f418_isr_1101475_8.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a28b9446ed1ae96b3c41735cdb3bd3d with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 0a80a271a1b126833fd044793425c4f1 parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d296b2fa-eb6c-43aa-a78c-d44c0caaaf38_characterizeImage_1101472_8.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d296b2fa-eb6c-43aa-a78c-d44c0caaaf38_characterizeImage_1101472_8.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/519a1485-20f9-4022-96cf-6dde2538fc08_isr_1101472_8.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/519a1485-20f9-4022-96cf-6dde2538fc08_isr_1101472_8.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2dd32d7e07366168ffc09c9667f59b4 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash 935c1eeb1f6de917767f8dcdd0420494 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609be340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4e3fe2f6-ba64-4024-b449-e185004e6174_characterizeImage_1101472_32.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4e3fe2f6-ba64-4024-b449-e185004e6174_characterizeImage_1101472_32.stderr parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1275757b-3246-445c-bb09-a1d841c84a6c_isr_1101472_32.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1275757b-3246-445c-bb09-a1d841c84a6c_isr_1101472_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:46 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b755cad5ffc0669373c7f5da56f65bc4 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash bf79f52df4cda9e1bf26a8a8c98d2dc8 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b76c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/138bde30-b0bc-4618-9795-9910de1c2513_characterizeImage_1101466_28.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/138bde30-b0bc-4618-9795-9910de1c2513_characterizeImage_1101466_28.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/9f7a509f-9388-429c-a19a-56da11502747_isr_1101466_28.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/9f7a509f-9388-429c-a19a-56da11502747_isr_1101466_28.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e21ea3d7936b20b0deb4b6b2a781bad with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 9c13058c2d62e86247a949d9cf43c313 parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609be160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e953d57e-c07d-4457-90fd-8134696a2907_characterizeImage_1101466_54.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e953d57e-c07d-4457-90fd-8134696a2907_characterizeImage_1101466_54.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/048d1933-a116-43ff-ab59-3bd8bcabfd50_isr_1101466_54.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/048d1933-a116-43ff-ab59-3bd8bcabfd50_isr_1101466_54.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0832f6aa68713bd558a5d816ca64d92 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash 75596326f1bd4a4eb7ef9d659ad3ade0 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609be980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/08b30b93-aca2-4d1d-b260-2612e5b9f698_characterizeImage_1101466_20.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/08b30b93-aca2-4d1d-b260-2612e5b9f698_characterizeImage_1101466_20.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/54e68ab2-258e-4fc8-bbb2-107a3a342710_isr_1101466_20.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/54e68ab2-258e-4fc8-bbb2-107a3a342710_isr_1101466_20.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49b7d22df1684b3d8864589efec13f5b with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash bfe637077da18149271e992fbd4454a6 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b777e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4911fa5d-1094-4f14-835b-a2cf632e94fd_characterizeImage_1101472_22.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4911fa5d-1094-4f14-835b-a2cf632e94fd_characterizeImage_1101472_22.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/c8394b16-9c40-488f-81ec-513b56672030_isr_1101472_22.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/c8394b16-9c40-488f-81ec-513b56672030_isr_1101472_22.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e582b7decac27e580a2b2a7734688440 with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash f871640ad9a9163a7853d05c98619f5e parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/5355e111-3c40-4608-9108-42210a716f0c_characterizeImage_1101466_56.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/5355e111-3c40-4608-9108-42210a716f0c_characterizeImage_1101466_56.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/d1465afb-35cd-4b3a-b0bf-999006a86a40_isr_1101466_56.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/d1465afb-35cd-4b3a-b0bf-999006a86a40_isr_1101466_56.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 657c35f87c8f0462623445cb5a5b8b02 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash addd529ac0d62b4529563da48ac47703 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b76f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/ae21c014-63dc-46fd-bc2d-e3fd3c08852d_characterizeImage_1101469_59.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/ae21c014-63dc-46fd-bc2d-e3fd3c08852d_characterizeImage_1101469_59.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/61bdb2ac-1127-49c0-9c31-bcf34c952631_isr_1101469_59.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/61bdb2ac-1127-49c0-9c31-bcf34c952631_isr_1101469_59.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a775c076432c84a4c4efa81e3c29158 with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 82f60a348f9f7d05acfd0e4a274c8a8b parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/79fdcb8d-5720-41e7-bfee-6a4db6a4f7de_characterizeImage_1101466_31.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/79fdcb8d-5720-41e7-bfee-6a4db6a4f7de_characterizeImage_1101466_31.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/4dcacf51-f388-43a8-bfb2-aa492bf5f7c8_isr_1101466_31.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/4dcacf51-f388-43a8-bfb2-aa492bf5f7c8_isr_1101466_31.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39250b82dc14c6c0e3b94adcf13c33fd with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash de12bf27d045bcf716024696369f6cfa parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/912a9589-7166-4b84-a638-2ebc8a3cc84f_characterizeImage_1101472_20.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/912a9589-7166-4b84-a638-2ebc8a3cc84f_characterizeImage_1101472_20.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/fa40109a-69a8-44e3-96d4-842f8e180e17_isr_1101472_20.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/fa40109a-69a8-44e3-96d4-842f8e180e17_isr_1101472_20.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce14ad537c5f56bedb4c282bd1b764ee with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash ef427fb7819345d8ea635ac0c3a34429 parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b765c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/9ae22180-cc92-41bb-9b99-fbb49ae5304a_characterizeImage_1101466_11.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/9ae22180-cc92-41bb-9b99-fbb49ae5304a_characterizeImage_1101466_11.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/bfdebce8-4fc3-4fbe-b5c5-a745ceb1763d_isr_1101466_11.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/bfdebce8-4fc3-4fbe-b5c5-a745ceb1763d_isr_1101466_11.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 817802d99416aaeef551f0f3cfbb3571 with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash de5e78a17cfbc8266ceb85c72901aaf1 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b768e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/17ba1955-00cb-447e-a5ef-a558f4ec61f1_characterizeImage_1101466_10.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/17ba1955-00cb-447e-a5ef-a558f4ec61f1_characterizeImage_1101466_10.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/7b496468-b7ec-4c65-8e8e-ced7c9efa109_isr_1101466_10.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/7b496468-b7ec-4c65-8e8e-ced7c9efa109_isr_1101466_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:47 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c6b534a550bf8388ef1a27d4b98bcff with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 446b390adfa937487a383faa5bfab2b0 parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b74220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/01be6faf-3406-4ae6-a019-2de1ae57f5b5_characterizeImage_1101475_62.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/01be6faf-3406-4ae6-a019-2de1ae57f5b5_characterizeImage_1101475_62.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/2cb85398-5e88-4dac-be16-da478721bf03_isr_1101475_62.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/2cb85398-5e88-4dac-be16-da478721bf03_isr_1101475_62.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c0fece2925b7660adacd98fcf0bef7d with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 71df7c13a8a53655de63e98526402d56 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bf4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/327665bb-94fb-4680-82d2-9f5c7f060273_characterizeImage_1101466_5.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/327665bb-94fb-4680-82d2-9f5c7f060273_characterizeImage_1101466_5.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/01b759bc-77b1-4c4e-8efd-e309af887303_isr_1101466_5.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/01b759bc-77b1-4c4e-8efd-e309af887303_isr_1101466_5.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10aa176d75fde3013a8288bc836018f6 with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 294cf19d3b8062f7cc9f0775772afba6 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b762a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/cb168b21-0cd0-4727-8800-9bc1b9aa8bc7_characterizeImage_1101472_9.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/cb168b21-0cd0-4727-8800-9bc1b9aa8bc7_characterizeImage_1101472_9.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/259c10a0-5e48-4f4e-b0b3-a92a8390a1f1_isr_1101472_9.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/259c10a0-5e48-4f4e-b0b3-a92a8390a1f1_isr_1101472_9.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a581c508d1c64756832735d0de847a3 with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 4978e3529088660dd51ed471cd0f098b parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/a8825bcc-a28b-40ca-9f20-7c286030dde0_characterizeImage_1101466_44.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/a8825bcc-a28b-40ca-9f20-7c286030dde0_characterizeImage_1101466_44.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/c43793a9-4b8e-4ec4-92b1-bdf080cea6cf_isr_1101466_44.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/c43793a9-4b8e-4ec4-92b1-bdf080cea6cf_isr_1101466_44.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d72ce0a0aebcccf0009f775804de3f5d with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 7ceed0d37a716f8c7531a3b9549a90bf parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b74ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/f9794297-4986-4d71-8ae9-66f09f45752c_characterizeImage_1101466_24.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/f9794297-4986-4d71-8ae9-66f09f45752c_characterizeImage_1101466_24.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/ac1b6702-dc04-434b-89ef-b898c525620a_isr_1101466_24.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/ac1b6702-dc04-434b-89ef-b898c525620a_isr_1101466_24.stderr parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7352bc6385222680cb91b0cd62031100 with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash f5f27e3125af0805b52b7622181fdfba parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a072e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/9aaf079c-9de7-4c31-a17e-0c01c4e03767_characterizeImage_1101475_23.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/9aaf079c-9de7-4c31-a17e-0c01c4e03767_characterizeImage_1101475_23.stderr parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/d4c2a2b6-c193-4e37-921b-e507fefb2c34_isr_1101475_23.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/d4c2a2b6-c193-4e37-921b-e507fefb2c34_isr_1101475_23.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca0f828e826ebd783e68ae27b51def43 with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash 1ed2ace7f4307839336ab16b09949170 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/f6cf8b98-8b9b-4284-bc24-52ceb3a3ed06_characterizeImage_1101469_44.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/f6cf8b98-8b9b-4284-bc24-52ceb3a3ed06_characterizeImage_1101469_44.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/286056e5-48c8-4def-a3ee-3f45aa3825b1_isr_1101469_44.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/286056e5-48c8-4def-a3ee-3f45aa3825b1_isr_1101469_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:48 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8100f27a4087e7833945a810bc2d0545 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 14e397b7660a6b837fc772f38b143278 parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a068e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/02d90e95-bf7a-4263-8e77-441c3de63973_characterizeImage_1101472_50.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/02d90e95-bf7a-4263-8e77-441c3de63973_characterizeImage_1101472_50.stderr parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/753d9d3c-8ab5-44d5-b2a5-19b778e95730_isr_1101472_50.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/753d9d3c-8ab5-44d5-b2a5-19b778e95730_isr_1101472_50.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 221bae2aca17d7d79cd4e079361c632a with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash d3dbd069d7b4cb5c3e39662981882c15 parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609be660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/ed2db7f7-40b3-4c67-86cb-2194a072bab5_characterizeImage_1101472_55.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/ed2db7f7-40b3-4c67-86cb-2194a072bab5_characterizeImage_1101472_55.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1b12dab2-3c5f-418a-b13a-cabedb66f10c_isr_1101472_55.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1b12dab2-3c5f-418a-b13a-cabedb66f10c_isr_1101472_55.stderr parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f768548ff7e3dec2c6f684eb376c29e5 with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash ea3feb77119b448074418b6b84b0f784 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/17059bde-0e36-49e7-ae34-dd4ec916e45b_characterizeImage_1101466_13.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/17059bde-0e36-49e7-ae34-dd4ec916e45b_characterizeImage_1101466_13.stderr parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b1315bb7-4d18-4233-9db2-01c4f96a6c99_isr_1101466_13.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b1315bb7-4d18-4233-9db2-01c4f96a6c99_isr_1101466_13.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64aab6190e8b3612bf6fb5cba3927643 with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 1567b81f8cc1468f71e6c303437c6636 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a049a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c0ea04aa-a9ef-4f3f-beae-feee370cb26d_characterizeImage_1101475_11.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c0ea04aa-a9ef-4f3f-beae-feee370cb26d_characterizeImage_1101475_11.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/cd05aa6c-5a88-4472-a6e7-2a985dfe51d5_isr_1101475_11.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/cd05aa6c-5a88-4472-a6e7-2a985dfe51d5_isr_1101475_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:48 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17efa2f352b3841c46cdb2349b1834d9 with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash bacea6909e9dc4325c6bd819b698d734 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/0101d367-2781-466f-8608-bc09e91e4fb8_characterizeImage_1101475_3.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/0101d367-2781-466f-8608-bc09e91e4fb8_characterizeImage_1101475_3.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c8f2ca01-94a7-4439-a491-43a3b20cdb24_isr_1101475_3.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c8f2ca01-94a7-4439-a491-43a3b20cdb24_isr_1101475_3.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e17ed9f540ae0799124c5c98da2d439 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 039b784095960f6d5de2f90e5925aa52 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a05620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/be62030f-d331-491f-aab4-a343704a5125_characterizeImage_1101469_31.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/be62030f-d331-491f-aab4-a343704a5125_characterizeImage_1101469_31.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/cfa31fbc-0513-46e6-a014-a74dfb93265e_isr_1101469_31.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/cfa31fbc-0513-46e6-a014-a74dfb93265e_isr_1101469_31.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d01211542f413e5dcf82aca637c5bfe6 with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash b9277ba2f6e00b6b52e6b7680b9b7e66 parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b747c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/70db0906-bcc8-461b-8c59-b0f6257e9ea8_characterizeImage_1101472_59.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/70db0906-bcc8-461b-8c59-b0f6257e9ea8_characterizeImage_1101472_59.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/5c3416cc-7d7f-4355-86f3-2f1bf67441c0_isr_1101472_59.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/5c3416cc-7d7f-4355-86f3-2f1bf67441c0_isr_1101472_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 PD 17226237 PD 17226238 PD 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:49 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 839fea963d7b364e0f568cacf8f41cb2 with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 98b7cdc8dab40c78f530106c3b57b4e9 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/87bfe706-b586-439a-9923-b8e8edcc2b8b_characterizeImage_1101466_55.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/87bfe706-b586-439a-9923-b8e8edcc2b8b_characterizeImage_1101466_55.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/acd8c3b1-9e6f-4964-b6ef-6983b246a21c_isr_1101466_55.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/acd8c3b1-9e6f-4964-b6ef-6983b246a21c_isr_1101466_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 PD 17226237 PD 17226238 PD 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:49 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e7269fba3a763c02122c5d8032821e3 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 0674deb369fe62c91b1bfb957566897b parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a05300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c132514c-76fb-4e82-97b7-bbc23d0aafe3_characterizeImage_1101475_56.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c132514c-76fb-4e82-97b7-bbc23d0aafe3_characterizeImage_1101475_56.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/fbb45ccd-7463-4d40-8fe4-1f7311f47b7e_isr_1101475_56.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/fbb45ccd-7463-4d40-8fe4-1f7311f47b7e_isr_1101475_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 PD 17226237 PD 17226238 PD 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:50 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f76f3b23315c0454ea5ce9e2e9a2f18 with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 125f56a243f6568124b40788942df0b1 parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/1fe46efa-688c-4c96-bec0-ae299db260a1_characterizeImage_1101466_36.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/1fe46efa-688c-4c96-bec0-ae299db260a1_characterizeImage_1101466_36.stderr parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b37f485a-6f98-4103-826e-4cae6c601e1b_isr_1101466_36.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/b37f485a-6f98-4103-826e-4cae6c601e1b_isr_1101466_36.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7b4f3c3a557a9d5ae209175b0047f23 with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash e3627242cf0419c89a694da7345b8873 parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/76b7c889-006a-4a42-8110-115ff2bc4d67_characterizeImage_1101469_42.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/76b7c889-006a-4a42-8110-115ff2bc4d67_characterizeImage_1101469_42.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/b3c0d116-926a-4040-9302-00163234cb7e_isr_1101469_42.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/b3c0d116-926a-4040-9302-00163234cb7e_isr_1101469_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 PD 17226237 PD 17226238 PD 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:50 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 PD 17226237 PD 17226238 PD 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:50 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 PD 17226237 PD 17226238 PD 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 59/3 running/pending blocks, and 81 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:02:51 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:02:51 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f683cefc93e4f654890a25fe4473b1bf with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 7b9c64b535c4263f78a8f06f6a72ae65 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a8bd12b7-946e-4015-84ca-202a827297d8_characterizeImage_1101475_59.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a8bd12b7-946e-4015-84ca-202a827297d8_characterizeImage_1101475_59.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/d223ba77-1db0-4fa2-8848-22cfb1fcc02d_isr_1101475_59.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/d223ba77-1db0-4fa2-8848-22cfb1fcc02d_isr_1101475_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:51 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56ef12314aca0d3eccf11514936bed0d with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 7c59eba3cd2f6e901c8bfee58ac0649a parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8f528bae-823c-4669-a83f-423304c44523_characterizeImage_1101472_2.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8f528bae-823c-4669-a83f-423304c44523_characterizeImage_1101472_2.stderr parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/4971ef5b-df21-4cca-87f8-d33a29a51fff_isr_1101472_2.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/4971ef5b-df21-4cca-87f8-d33a29a51fff_isr_1101472_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:51 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:52 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:52 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:52 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:52 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2698b69d4eb6cbcbbcf240166646136d with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash 22b3e9a98a18e0cfc91ec36822c7c082 parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a06fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d819302c-3880-463e-a951-536454af31a3_characterizeImage_1101472_21.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d819302c-3880-463e-a951-536454af31a3_characterizeImage_1101472_21.stderr parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/26a5f816-ea71-4dfb-8675-d647330d362e_isr_1101472_21.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/26a5f816-ea71-4dfb-8675-d647330d362e_isr_1101472_21.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8181a6a7fb45f8e32d6c05c9eb7d7bce with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 8a40e02e7c1ed2b35b8228540b66f2bc parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a05a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:53 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/c706ce05-da4c-4940-b1e3-1e90357a1a22_characterizeImage_1101472_30.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/c706ce05-da4c-4940-b1e3-1e90357a1a22_characterizeImage_1101472_30.stderr parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/2b9a3330-0593-423f-9c71-b0f56d875bc3_isr_1101472_30.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/2b9a3330-0593-423f-9c71-b0f56d875bc3_isr_1101472_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:53 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:53 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f7adb8a02fa9c26def6a058081e903c with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 7b0a04fa4aee5c0d15c5ef95e3dfac8a parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/094998b6-3adf-4aa8-abf0-c0cbbadb6985_characterizeImage_1101469_49.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/094998b6-3adf-4aa8-abf0-c0cbbadb6985_characterizeImage_1101469_49.stderr parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/59d33119-980b-4a32-87c3-ed98a9939cb7_isr_1101469_49.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/59d33119-980b-4a32-87c3-ed98a9939cb7_isr_1101469_49.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7460f28ebd9c275cefa9a9b7fcb95b77 with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash 2c30bba3bf24d6d6787d87772450e201 parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/46a4e94f-a547-4de9-b9db-bb01375dedf9_characterizeImage_1101469_21.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/46a4e94f-a547-4de9-b9db-bb01375dedf9_characterizeImage_1101469_21.stderr parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/46dd967e-a4a9-43ae-a0ff-0430602fa0d7_isr_1101469_21.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/46dd967e-a4a9-43ae-a0ff-0430602fa0d7_isr_1101469_21.stderr parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79292d2552ff6016fbbd0853e3c5f352 with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash c2f8e336b7ca160285845ae81dca7a80 parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/78dd6562-5636-46b5-bc11-1a5ec69bdeb2_characterizeImage_1101475_15.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/78dd6562-5636-46b5-bc11-1a5ec69bdeb2_characterizeImage_1101475_15.stderr parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/9fa98f43-9d08-4610-9c3f-a2217b54c5ab_isr_1101475_15.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/9fa98f43-9d08-4610-9c3f-a2217b54c5ab_isr_1101475_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:53 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26fbca2a20fe16e6005402b8141d97dc with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 57220570230170300cf8bee13a9ea07a parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/02e9c2e7-f7d7-4571-803e-6cf3dbfbc386_characterizeImage_1101472_10.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/02e9c2e7-f7d7-4571-803e-6cf3dbfbc386_characterizeImage_1101472_10.stderr parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/8de921e0-2a90-4471-b1e1-df321e2d82d2_isr_1101472_10.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/8de921e0-2a90-4471-b1e1-df321e2d82d2_isr_1101472_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:54 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1006fb1efd45c9fefa7cf0125af3d65f with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 5e38e7219035cbc06e84fb94e1396b58 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/fa9c43b7-1dbd-4024-8f4c-9f0b3f4467ab_characterizeImage_1101472_5.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/fa9c43b7-1dbd-4024-8f4c-9f0b3f4467ab_characterizeImage_1101472_5.stderr parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a236a505-f60a-49f2-a89a-1001474786e5_isr_1101472_5.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a236a505-f60a-49f2-a89a-1001474786e5_isr_1101472_5.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 351cf9e39a042be05cbf2e848662cda3 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash e78e931239f603b32d713fb75baef07f parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a05f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c6d41c88-73e8-445f-b15c-6276ecf551c8_characterizeImage_1101469_10.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c6d41c88-73e8-445f-b15c-6276ecf551c8_characterizeImage_1101469_10.stderr parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/01624a47-06dc-4f6f-a74b-4a8505a45aff_isr_1101469_10.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/01624a47-06dc-4f6f-a74b-4a8505a45aff_isr_1101469_10.stderr parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50cf4de739d876dfe8adedf4d3a98171 with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 01e4d546b48958a0d5658d118ff2c833 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/abc03a38-f803-481d-b73a-c273cbd6bbda_characterizeImage_1101469_35.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/abc03a38-f803-481d-b73a-c273cbd6bbda_characterizeImage_1101469_35.stderr parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/72c75c1f-e77b-4eca-a52d-c4a768226772_isr_1101469_35.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/72c75c1f-e77b-4eca-a52d-c4a768226772_isr_1101469_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:54 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a4237febe04b02bec5705d689f4a1b9 with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 802e1dee81f7c6bb0f3a8cf18a2c7876 parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a06520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7588e5ec-5039-4385-b4a1-4dfe6bc09282_characterizeImage_1101466_40.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7588e5ec-5039-4385-b4a1-4dfe6bc09282_characterizeImage_1101466_40.stderr parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/666722b0-5ff7-40af-83d0-444b7e601058_isr_1101466_40.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/666722b0-5ff7-40af-83d0-444b7e601058_isr_1101466_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:54 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d9cd4da7af128b55d045ddbecd80916 with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 3ad7b58246fa3dcdf87bdb4612cbfeda parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/56b73c35-0fbc-4339-bb4c-09e87b4c58b5_characterizeImage_1101472_57.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/56b73c35-0fbc-4339-bb4c-09e87b4c58b5_characterizeImage_1101472_57.stderr parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/00ffac42-e0c4-477a-a4c5-81e49b2dea9d_isr_1101472_57.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/00ffac42-e0c4-477a-a4c5-81e49b2dea9d_isr_1101472_57.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 524eecb827dae471ba2bd19b125e689b with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash a63136167703d5c7ca4e267928e79c86 parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/cc01c6fe-c1aa-4b92-b6cc-855a9a262deb_characterizeImage_1101475_20.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/cc01c6fe-c1aa-4b92-b6cc-855a9a262deb_characterizeImage_1101475_20.stderr parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/9cf9d04d-e85e-4009-b3e9-1d9580468e07_isr_1101475_20.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/9cf9d04d-e85e-4009-b3e9-1d9580468e07_isr_1101475_20.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry addf053f379917d43d028fc9b849bd66 with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 9411714248360c11fce0129036ad6611 parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/7f2df31e-9dec-48d2-9261-d2307ddfa367_characterizeImage_1101472_7.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/7f2df31e-9dec-48d2-9261-d2307ddfa367_characterizeImage_1101472_7.stderr parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/cd40b11c-9f27-4b1b-8e9e-61e027cc7d73_isr_1101472_7.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/cd40b11c-9f27-4b1b-8e9e-61e027cc7d73_isr_1101472_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:54 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 551c6ee8240504a97078a8336d42e33f with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 1d4525dfe935e3c3dec2709f347577d0 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7c18b06e-733b-4915-ac61-ad00b68f3015_characterizeImage_1101466_23.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7c18b06e-733b-4915-ac61-ad00b68f3015_characterizeImage_1101466_23.stderr parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/cd601cbd-078a-4258-9203-c6afa0dc6355_isr_1101466_23.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/cd601cbd-078a-4258-9203-c6afa0dc6355_isr_1101466_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:55 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3fdbf53470d9c46b916a1b470747111 with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash 5596728118ec0dba43ddf84a1a6d4081 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a06ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/b2345076-cd56-4ef7-8f5c-898e202a9672_characterizeImage_1101472_35.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/b2345076-cd56-4ef7-8f5c-898e202a9672_characterizeImage_1101472_35.stderr parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/95197869-5337-406c-a112-b085d9f3f284_isr_1101472_35.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/95197869-5337-406c-a112-b085d9f3f284_isr_1101472_35.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29fd6301767642f53a9c19e20a4a8397 with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash eba4c069785fce6140e6206dcc3f4c41 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a05e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/255a48e2-0aa9-4326-b6ed-7b12518b2f4a_characterizeImage_1101469_55.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/255a48e2-0aa9-4326-b6ed-7b12518b2f4a_characterizeImage_1101469_55.stderr parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/e631e2b3-00c1-4fac-8d57-4f2bffc6f83d_isr_1101469_55.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/e631e2b3-00c1-4fac-8d57-4f2bffc6f83d_isr_1101469_55.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 957c62b8bceba8d0a285d93555dc29b3 with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash 74f8056d8a1bb802c70599beaf29b218 parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:55 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/7b86f3de-b0d4-4a77-aa93-525453f557af_characterizeImage_1101475_10.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/7b86f3de-b0d4-4a77-aa93-525453f557af_characterizeImage_1101475_10.stderr parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/04991cd6-7e68-4528-91c7-a7d2bdc83d7f_isr_1101475_10.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/04991cd6-7e68-4528-91c7-a7d2bdc83d7f_isr_1101475_10.stderr parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 897d80d1f9ec041528b8e4169321aede with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash 59c8996421a58ae9101ddccaab2b027e parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/088bbc04-438b-40e2-8ec0-2e129cb0a1dd_characterizeImage_1101475_30.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/088bbc04-438b-40e2-8ec0-2e129cb0a1dd_characterizeImage_1101475_30.stderr parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/4769ee8f-f69e-41c1-8a51-2f3729af687f_isr_1101475_30.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/4769ee8f-f69e-41c1-8a51-2f3729af687f_isr_1101475_30.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb0569782cad957d211f395173206bef with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash b714184d1fc8d48d76f6feb89de7ba69 parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/62830edd-3a21-4165-8fda-50d4a4f85b5a_characterizeImage_1101472_25.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/62830edd-3a21-4165-8fda-50d4a4f85b5a_characterizeImage_1101472_25.stderr parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/03c4a7c3-5968-441e-b92b-a2b3688195e7_isr_1101472_25.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/03c4a7c3-5968-441e-b92b-a2b3688195e7_isr_1101472_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:55 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:55 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:55 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4034ff455b35a7436af517ddfd65e5ca with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash ab8d28e9d9958c0c694037d8f8eaa032 parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/dad1cbc9-8497-4c32-becf-6d3cd67c6c4e_characterizeImage_1101466_59.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/dad1cbc9-8497-4c32-becf-6d3cd67c6c4e_characterizeImage_1101466_59.stderr parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/86ce974f-54c1-42bc-8f17-a821b6dd9031_isr_1101466_59.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/86ce974f-54c1-42bc-8f17-a821b6dd9031_isr_1101466_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:56 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 503b25ac5e487a8bbd51414869e9374f with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash c8f3832366b9d1e3f127efc62adabe99 parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/96b4e15f-b2fd-4520-8edc-4183a26342b4_characterizeImage_1101475_43.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/96b4e15f-b2fd-4520-8edc-4183a26342b4_characterizeImage_1101475_43.stderr parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/698167f8-167c-40c2-947c-f4c1a2175fac_isr_1101475_43.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/698167f8-167c-40c2-947c-f4c1a2175fac_isr_1101475_43.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc78e1cbf1c0c2704e0e78fb1d559b24 with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 11f9593219ffc9adf5a701ac41913a5f parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/538a3462-ee07-465c-8ee3-e2ff7b8ca249_characterizeImage_1101469_54.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/538a3462-ee07-465c-8ee3-e2ff7b8ca249_characterizeImage_1101469_54.stderr parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/6bae0d28-e020-4754-a418-968f259d4b7d_isr_1101469_54.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/6bae0d28-e020-4754-a418-968f259d4b7d_isr_1101469_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:56 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4274ece609c02bee6805572c3bf90fa8 with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash 4164cccb4001a40443feb32e509c691c parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/8d1193e2-641b-45a3-a6c1-da4fcba36527_characterizeImage_1101466_38.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/8d1193e2-641b-45a3-a6c1-da4fcba36527_characterizeImage_1101466_38.stderr parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/bad96ae4-85c5-498a-9591-184f19a169f5_isr_1101466_38.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/bad96ae4-85c5-498a-9591-184f19a169f5_isr_1101466_38.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e62ad23dc5add36891cd23280736ea96 with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash b700c664a90bf0b1ee5c70b31aaa1c55 parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a50a4147-fe8e-4a7e-96ae-7135836c071e_characterizeImage_1101475_4.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a50a4147-fe8e-4a7e-96ae-7135836c071e_characterizeImage_1101475_4.stderr parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/5b227a2b-b44d-4ae1-8085-18f3071277ac_isr_1101475_4.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/5b227a2b-b44d-4ae1-8085-18f3071277ac_isr_1101475_4.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0d5108342bdedcea3335bbc26a13c35 with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash f35ae243f8d3517191e4e1975347be52 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/ca1ee52b-97c8-4e8b-bb0c-7e01e5143e51_characterizeImage_1101472_26.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/ca1ee52b-97c8-4e8b-bb0c-7e01e5143e51_characterizeImage_1101472_26.stderr parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a0b01fcd-46f5-4a35-ac83-3da25e2be4b0_isr_1101472_26.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a0b01fcd-46f5-4a35-ac83-3da25e2be4b0_isr_1101472_26.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df1476da419bd59ab9b950038c8776a2 with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash a79291baacada01d90943f7f272f18f9 parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/3c707758-39b6-4827-99c6-6af94e8b8957_characterizeImage_1101472_37.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/3c707758-39b6-4827-99c6-6af94e8b8957_characterizeImage_1101472_37.stderr parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/3dd4d308-3015-4cf9-b641-1c878a747ec5_isr_1101472_37.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/3dd4d308-3015-4cf9-b641-1c878a747ec5_isr_1101472_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:57 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:02:59 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f638c2b0a678cf94e9f23601d384559b with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 7ac934a1474c6dab12037277a6047b06 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/f79e9085-e809-412a-8cc9-eaaf3a9c4b0a_characterizeImage_1101469_36.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/f79e9085-e809-412a-8cc9-eaaf3a9c4b0a_characterizeImage_1101469_36.stderr parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/73b55365-0b76-409a-806a-b8860df409a6_isr_1101469_36.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/73b55365-0b76-409a-806a-b8860df409a6_isr_1101469_36.stderr parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7cad2914fe7fa5c1d9ced7701406c2f with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash 0922f679b503b503ee3c02436e2b1c57 parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/5f7f32c2-9567-4f1e-8623-8676cf680541_characterizeImage_1101469_4.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/5f7f32c2-9567-4f1e-8623-8676cf680541_characterizeImage_1101469_4.stderr parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/8ba93521-acb0-409a-9320-b4c9286529f7_isr_1101469_4.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/8ba93521-acb0-409a-9320-b4c9286529f7_isr_1101469_4.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf842d590ef6b3949240886f54b4a401 with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 02091856f499129ec29d1cacde274965 parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/d1411d97-da67-4d6e-802d-2d6f766424e6_characterizeImage_1101469_39.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/d1411d97-da67-4d6e-802d-2d6f766424e6_characterizeImage_1101469_39.stderr parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9b2f46c0-3eec-4b1b-ab0a-c06f673f3dee_isr_1101469_39.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/9b2f46c0-3eec-4b1b-ab0a-c06f673f3dee_isr_1101469_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:00 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 981b531c6a41f2db8526d8df40ae9224 with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 8b48e92cc58d62b9d9330aabe05342ea parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/762e74c3-40d8-46ca-ac75-aa33436361f4_characterizeImage_1101466_8.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/762e74c3-40d8-46ca-ac75-aa33436361f4_characterizeImage_1101466_8.stderr parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/3b4dc638-ad81-4362-bd3c-4daa667ee3cd_isr_1101466_8.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/3b4dc638-ad81-4362-bd3c-4daa667ee3cd_isr_1101466_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:01 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3cb84509a7966e21a72e3cd17d3754e with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash b6e2527911b37feb2a0d02de86d89f98 parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/21cddd1d-8494-4d8d-965f-12e413387de7_characterizeImage_1101469_1.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/21cddd1d-8494-4d8d-965f-12e413387de7_characterizeImage_1101469_1.stderr parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/5cb2c4b1-182a-4563-8479-eb91d70f0baa_isr_1101469_1.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/5cb2c4b1-182a-4563-8479-eb91d70f0baa_isr_1101469_1.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b2bfa48abd93f3050087682aec3068c with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash c0474a85fc4e8c91821adb378376ffba parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ef920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/8d519ac7-61ca-4bf9-adcc-30195ea3a889_characterizeImage_1101466_15.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/8d519ac7-61ca-4bf9-adcc-30195ea3a889_characterizeImage_1101466_15.stderr parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/9756227c-16c9-44d5-bdfb-bbc25d125904_isr_1101466_15.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/9756227c-16c9-44d5-bdfb-bbc25d125904_isr_1101466_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:04 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 734464c277850245484ac089d22c33ef with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 3b7e8d58b4fc7ae7e3cbb7f8a9f4147b parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ecd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4c0ddab0-a20f-4dd5-a3be-39f8d5e69664_characterizeImage_1101472_47.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4c0ddab0-a20f-4dd5-a3be-39f8d5e69664_characterizeImage_1101472_47.stderr parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d368f1d8-e041-43ba-be26-f9377fd189c0_isr_1101472_47.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/d368f1d8-e041-43ba-be26-f9377fd189c0_isr_1101472_47.stderr parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 228232d67eda0731a2682deaf421c469 with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash d2ae1dee939f2c1a478247ad135615e6 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/75859e87-bd40-4928-8508-bcc3d5ee2213_characterizeImage_1101466_7.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/75859e87-bd40-4928-8508-bcc3d5ee2213_characterizeImage_1101466_7.stderr parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/45256cd9-25a1-4585-a5f1-5701dd4baaaf_isr_1101466_7.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/45256cd9-25a1-4585-a5f1-5701dd4baaaf_isr_1101466_7.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01736a1a9fcdea8dcd947d083c1f7224 with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash e7130e4d1681d3f911444f3a16eecea3 parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ec720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/72d36d32-5d23-4ec3-9237-abbf3c14d089_characterizeImage_1101469_53.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/72d36d32-5d23-4ec3-9237-abbf3c14d089_characterizeImage_1101469_53.stderr parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/957e324c-4716-4f42-9ea7-dab325f84d74_isr_1101469_53.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/957e324c-4716-4f42-9ea7-dab325f84d74_isr_1101469_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:06 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 992b2ec9360bce2a2277c99e70d1a86a with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 29433438bb2f0774455958ef370a00cc parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ef600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/930a545f-e962-44cd-9a84-e7005d50a790_characterizeImage_1101475_55.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/930a545f-e962-44cd-9a84-e7005d50a790_characterizeImage_1101475_55.stderr parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/52e68085-dde9-46b3-931e-b474fecab22f_isr_1101475_55.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/52e68085-dde9-46b3-931e-b474fecab22f_isr_1101475_55.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b2df008a95af81d9acb64d26fcd58c0 with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 54e9a440d4a0819de97b3c80a98ba6df parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/348bc317-d6f8-4f8b-a807-8a5e0cf4fb61_characterizeImage_1101466_32.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/348bc317-d6f8-4f8b-a807-8a5e0cf4fb61_characterizeImage_1101466_32.stderr parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/2aeeb6b7-a80f-4190-b5c8-e6b445322a65_isr_1101466_32.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/2aeeb6b7-a80f-4190-b5c8-e6b445322a65_isr_1101466_32.stderr parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 767fe7d83f6710fc3fbcb5ad1fc4dfa0 with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 7a04e6ef6671d41beddca466c81d4317 parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/dc406187-6604-4929-a509-8f85f13a8794_characterizeImage_1101466_33.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/dc406187-6604-4929-a509-8f85f13a8794_characterizeImage_1101466_33.stderr parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/57646193-b9c7-4531-8853-19cd9d940deb_isr_1101466_33.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/57646193-b9c7-4531-8853-19cd9d940deb_isr_1101466_33.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 515e852524fd7b8c51c7b4bdc4afaeeb with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash 99e89c560dc7150cc8ca22e7b1174d61 parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ed080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/33553aa4-3b02-4610-9b80-4f1e3543fe10_characterizeImage_1101475_34.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/33553aa4-3b02-4610-9b80-4f1e3543fe10_characterizeImage_1101475_34.stderr parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/74b6eea8-8010-4b86-b94c-b7c834b0eba8_isr_1101475_34.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/74b6eea8-8010-4b86-b94c-b7c834b0eba8_isr_1101475_34.stderr parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a159192188c7b2f6a53085cefbdd96fc with result from task 681 parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:09 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 20959bd011166cefd1d8d98df727eb47 parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/ba710486-f66c-47f3-b5ae-2d6a0ce6a01b_characterizeImage_1101466_45.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/ba710486-f66c-47f3-b5ae-2d6a0ce6a01b_characterizeImage_1101466_45.stderr parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/966b2fa6-52fa-404a-acd6-65df7c484ca3_isr_1101466_45.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/966b2fa6-52fa-404a-acd6-65df7c484ca3_isr_1101466_45.stderr parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b24b0d915453a7fcf0819768658700dc with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash 97106baa6d443e2d82adbb3f5d87b59b parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/63e73a4e-50fb-41c8-a9cf-e10f6a4dea2d_characterizeImage_1101472_34.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/63e73a4e-50fb-41c8-a9cf-e10f6a4dea2d_characterizeImage_1101472_34.stderr parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1c2d8f73-a6ba-45a1-af63-b9e648f3d9f1_isr_1101472_34.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1c2d8f73-a6ba-45a1-af63-b9e648f3d9f1_isr_1101472_34.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01171106e968f6cdcb4eb0c0ee0ab954 with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 1830bcd77900e4722c0b8cec911bc3cc parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/63f662ea-585e-4dee-8824-f666c97740b7_characterizeImage_1101466_39.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/63f662ea-585e-4dee-8824-f666c97740b7_characterizeImage_1101466_39.stderr parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/887979dd-8f0f-47b9-bec2-8e38caaff152_isr_1101466_39.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/887979dd-8f0f-47b9-bec2-8e38caaff152_isr_1101466_39.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48571c8386e23a240bef511ec0a91d60 with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash b12b3c34f4f1f5ba0ba591657876544d parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c06c6598-337d-4f2f-b32e-5ed1b7d9cdfe_characterizeImage_1101469_6.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c06c6598-337d-4f2f-b32e-5ed1b7d9cdfe_characterizeImage_1101469_6.stderr parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/975f0522-8187-45e8-84ba-53dc24e5e024_isr_1101469_6.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/975f0522-8187-45e8-84ba-53dc24e5e024_isr_1101469_6.stderr parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7b324ad1f592042beb61b3ace9a13dd with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash d620bb2a45dded8bd0990f92c6d066c1 parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/66a6216b-a828-490c-9335-176ad9cfcfca_characterizeImage_1101469_16.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/66a6216b-a828-490c-9335-176ad9cfcfca_characterizeImage_1101469_16.stderr parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/ce53ed7a-6e11-4124-960a-4ecef76dedf5_isr_1101469_16.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/ce53ed7a-6e11-4124-960a-4ecef76dedf5_isr_1101469_16.stderr parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c88324af58a471a494a4e7d412fbb00 with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash 561be792900fdc6db8b3c71a7bfc4060 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608eca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/c0f8ad3f-91cb-432f-89d3-6fbe962e341a_characterizeImage_1101472_45.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/c0f8ad3f-91cb-432f-89d3-6fbe962e341a_characterizeImage_1101472_45.stderr parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1113f154-dd41-4aab-a295-1747ae018250_isr_1101472_45.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/1113f154-dd41-4aab-a295-1747ae018250_isr_1101472_45.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e09316219e7df10e8856ab281dacfc18 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 9fcedd3ddfd306e163b20d5d24245134 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/524ecf8d-1ef2-496e-86a1-9c1ac470db29_characterizeImage_1101469_24.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/524ecf8d-1ef2-496e-86a1-9c1ac470db29_characterizeImage_1101469_24.stderr parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/2f11d5fa-b2a8-4004-a14b-d9eee1a32efe_isr_1101469_24.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/2f11d5fa-b2a8-4004-a14b-d9eee1a32efe_isr_1101469_24.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 452760e81ef82e7ee4a2250b4a74ae0c with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash fa4cf43e79e37cf5f8229746143f6f84 parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ec400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/baa97767-9413-4ae0-86fb-9fe5ea018ab1_characterizeImage_1101469_29.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/baa97767-9413-4ae0-86fb-9fe5ea018ab1_characterizeImage_1101469_29.stderr parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/dccf0fad-11b3-46db-9d5d-68bf8c851887_isr_1101469_29.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/dccf0fad-11b3-46db-9d5d-68bf8c851887_isr_1101469_29.stderr parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adca080a3e3b233b7730a32e9dd6df65 with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 258326f4befeedbea04f82c3ece7d661 parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/b9faa86c-a60f-4f81-854f-e0c53abe1d96_characterizeImage_1101469_28.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/b9faa86c-a60f-4f81-854f-e0c53abe1d96_characterizeImage_1101469_28.stderr parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/ed7e42a5-26bc-4a90-ace2-44fdf553a2c5_isr_1101469_28.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/ed7e42a5-26bc-4a90-ace2-44fdf553a2c5_isr_1101469_28.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99d23c64eb1fbd95d50e49a03c03078a with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash d1dcd0f0f774312b59a1560a395901b2 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c65fa402-599a-4790-b575-f77a3e367532_characterizeImage_1101466_9.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c65fa402-599a-4790-b575-f77a3e367532_characterizeImage_1101466_9.stderr parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/c6a820cb-4ab0-4c56-b966-6a5b39b56491_isr_1101466_9.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/c6a820cb-4ab0-4c56-b966-6a5b39b56491_isr_1101466_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:10 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22d23aa72a50cff4413db19a5a97f904 with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 612c445a8a9bc9a880522fb6ad93ea72 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608efc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8eaf6abe-dd4e-4997-a835-5550655038d8_characterizeImage_1101475_42.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8eaf6abe-dd4e-4997-a835-5550655038d8_characterizeImage_1101475_42.stderr parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/6de27fed-97fc-44e8-a665-217ea162886f_isr_1101475_42.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/6de27fed-97fc-44e8-a665-217ea162886f_isr_1101475_42.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfd0103f8a078a2ea1f738bf80c9323d with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 5900145034421ee88bee60e25b01f1e1 parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/283b002b-8244-47bf-9aa8-ca08b51c929d_characterizeImage_1101475_46.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/283b002b-8244-47bf-9aa8-ca08b51c929d_characterizeImage_1101475_46.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/456ffadb-8b96-432a-bd43-7d38d1ac638d_isr_1101475_46.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/456ffadb-8b96-432a-bd43-7d38d1ac638d_isr_1101475_46.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1f44a75afe4ecb8b3cde144a762b52a with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 9c317d866f2e48b2400d8b5ebe01c47c parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b75a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/037ac4e7-ddad-4bfb-bb20-ef9a0852e338_characterizeImage_1101475_33.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/037ac4e7-ddad-4bfb-bb20-ef9a0852e338_characterizeImage_1101475_33.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/76c16be2-77c5-4641-9f41-f35c9dbd429b_isr_1101475_33.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/76c16be2-77c5-4641-9f41-f35c9dbd429b_isr_1101475_33.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77a78ff2a74b08f5ec7c16083c388cd1 with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 6d512f3b1a164efe722909651ef9447c parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b75620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/64c6b6e4-3eb0-4dae-a16c-11ff19fe6a87_characterizeImage_1101469_22.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/64c6b6e4-3eb0-4dae-a16c-11ff19fe6a87_characterizeImage_1101469_22.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/12e92dfd-da82-447b-a099-0fddb6f928da_isr_1101469_22.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/12e92dfd-da82-447b-a099-0fddb6f928da_isr_1101469_22.stderr parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ef3bfd4b22ea021e15329fc1ce7c1df with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 0d2380f0cf884289814966126dc048c3 parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/f0d1938e-ce4b-492e-bdfa-89f26dc3ab03_characterizeImage_1101466_58.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/f0d1938e-ce4b-492e-bdfa-89f26dc3ab03_characterizeImage_1101466_58.stderr parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/03ee38db-4b2c-4775-ae05-01ab3c42d354_isr_1101466_58.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/03ee38db-4b2c-4775-ae05-01ab3c42d354_isr_1101466_58.stderr parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e62cbccb4f0b34262f44d700dc683b10 with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 8e9629d88b769c253b4c005db74158fc parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/2db12c8b-c956-4a61-a5bc-ff20694be85d_characterizeImage_1101466_29.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/2db12c8b-c956-4a61-a5bc-ff20694be85d_characterizeImage_1101466_29.stderr parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/54f9a8bd-1cca-41dd-8c30-5901cc2b1025_isr_1101466_29.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/54f9a8bd-1cca-41dd-8c30-5901cc2b1025_isr_1101466_29.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7011b4b30f1e7362a71c5beed7938b7 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 47814b1df7108c88ce7fcb39d6cece94 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b74fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/85d52d3b-34f4-403b-b2e9-4842ab9f066d_characterizeImage_1101472_29.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/85d52d3b-34f4-403b-b2e9-4842ab9f066d_characterizeImage_1101472_29.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/070adc68-5037-4fe8-86b3-2d7216fe7566_isr_1101472_29.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/070adc68-5037-4fe8-86b3-2d7216fe7566_isr_1101472_29.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0566bb4fc5befd3879aa8a25b19d430 with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash f5339f5e31493474d14b08c838365687 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2e9aa646-860d-45ee-b987-a0bd13dd6b01_characterizeImage_1101469_56.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2e9aa646-860d-45ee-b987-a0bd13dd6b01_characterizeImage_1101469_56.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/15f98f60-3bce-490a-9ff6-1f5fcba30498_isr_1101469_56.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/15f98f60-3bce-490a-9ff6-1f5fcba30498_isr_1101469_56.stderr parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82b2a39015876ce19d122d094df7214c with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash f6ad6f92888e42143011df4b7c04c296 parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b76340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/5a07453c-101a-4b27-9f0a-f3fc7187456f_characterizeImage_1101472_46.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/5a07453c-101a-4b27-9f0a-f3fc7187456f_characterizeImage_1101472_46.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/10dd77ef-005e-492d-b0a0-53d5d599235c_isr_1101472_46.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/10dd77ef-005e-492d-b0a0-53d5d599235c_isr_1101472_46.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b942ab2e3444fcbcd8aff4b6b867c076 with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 1d5cbb1ebc6ea2ea1085ad7cd917403f parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/3096b2cc-96ee-4c23-970b-c3927271b530_characterizeImage_1101469_26.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/3096b2cc-96ee-4c23-970b-c3927271b530_characterizeImage_1101469_26.stderr parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/035f1484-9acf-434f-9f72-01b83708c48a_isr_1101469_26.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/035f1484-9acf-434f-9f72-01b83708c48a_isr_1101469_26.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6057c43be3f5c9fbe2c0a381a157d268 with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash 261987e7de076c5b3aeaf5f9efc82480 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/f85bd32a-edcc-45ae-8efc-6dae6b247d9c_characterizeImage_1101466_43.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/f85bd32a-edcc-45ae-8efc-6dae6b247d9c_characterizeImage_1101466_43.stderr parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/bc63409f-263f-4899-8fc3-ea1dd5372245_isr_1101466_43.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/bc63409f-263f-4899-8fc3-ea1dd5372245_isr_1101466_43.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e50f341328f906b347895891d3a30d7 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash c2ebb0b02c90f55d541642fa70c83ccb parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:11 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4ea3ad58-ffc5-4fce-9cb3-8cd8654c7995_characterizeImage_1101472_44.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4ea3ad58-ffc5-4fce-9cb3-8cd8654c7995_characterizeImage_1101472_44.stderr parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/9f6ce7a8-0443-4688-8ce0-6eee6efac2bb_isr_1101472_44.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/9f6ce7a8-0443-4688-8ce0-6eee6efac2bb_isr_1101472_44.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a192573e228427645f821eecfbb01bc with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash a43091f6b9a55e59054506fee7a70a97 parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b77b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/aae529ce-4359-4d6a-9abf-67e46e949fe2_characterizeImage_1101472_38.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/aae529ce-4359-4d6a-9abf-67e46e949fe2_characterizeImage_1101472_38.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/66f547e2-2cf4-4dc1-8256-bf96639266bb_isr_1101472_38.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/66f547e2-2cf4-4dc1-8256-bf96639266bb_isr_1101472_38.stderr parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee2ae28b388fc4a2c5c94015a416987d with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash 996eb8c6c7f9c9a612701395a991ea98 parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/769c713d-11cb-4d27-be90-7d91309a12e6_characterizeImage_1101472_53.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/769c713d-11cb-4d27-be90-7d91309a12e6_characterizeImage_1101472_53.stderr parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a1134e9e-fd84-4197-8eaf-13d27b7932ab_isr_1101472_53.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/a1134e9e-fd84-4197-8eaf-13d27b7932ab_isr_1101472_53.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2cf5da8066c3d5a682e223ad2d11599 with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash fd8661637b097f7d20eb83d6e5030f60 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b774c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/bc7f7478-6e0b-4aa1-a707-14e75a17a97f_characterizeImage_1101466_26.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/bc7f7478-6e0b-4aa1-a707-14e75a17a97f_characterizeImage_1101466_26.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/2e4d2491-3492-4f50-9710-d67738342247_isr_1101466_26.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/2e4d2491-3492-4f50-9710-d67738342247_isr_1101466_26.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8502eebde89b9c42585bdd850eca7bd5 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 726f46e0cb292d58bbb99740f388c258 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/040b5c50-d5fa-46b9-92e3-65952c61ea15_characterizeImage_1101466_22.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/040b5c50-d5fa-46b9-92e3-65952c61ea15_characterizeImage_1101466_22.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/25046fa8-f321-4893-8970-f9035bbcc078_isr_1101466_22.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/25046fa8-f321-4893-8970-f9035bbcc078_isr_1101466_22.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f62fffca0e4bf84d742f738f10adc9ac with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 46398e283ab7328883a29c3c0997d678 parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b74e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/90eaed79-86cb-405d-a71f-791c7790c04e_characterizeImage_1101466_4.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/90eaed79-86cb-405d-a71f-791c7790c04e_characterizeImage_1101466_4.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/469a2791-dbee-4471-8295-c02999033101_isr_1101466_4.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/469a2791-dbee-4471-8295-c02999033101_isr_1101466_4.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b490ba8c33b9c3e42f182ae3f9001472 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 5b62d86f38647429a660bcc4b7199479 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/44a9e101-8901-416e-a0d9-6bdb1e5347f2_characterizeImage_1101466_47.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/44a9e101-8901-416e-a0d9-6bdb1e5347f2_characterizeImage_1101466_47.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/98df82ef-c3e3-4b5e-8e3a-96e2ecaf71f8_isr_1101466_47.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/98df82ef-c3e3-4b5e-8e3a-96e2ecaf71f8_isr_1101466_47.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80f7fac2111600037f16b2e12090e851 with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash 3f8ff938eb24a46ad4bbd2af41581529 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b75da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/2ff0b43a-caea-4b48-9215-30260cfdf2b6_characterizeImage_1101472_60.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/2ff0b43a-caea-4b48-9215-30260cfdf2b6_characterizeImage_1101472_60.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/e45a0eb0-ee20-4cd3-8ef8-843b33dd6df0_isr_1101472_60.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101472/e45a0eb0-ee20-4cd3-8ef8-843b33dd6df0_isr_1101472_60.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb7ef51dd02b948e4c31d12afdaf58b0 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash 143883b13b47be2b2b5bca18f51bb14d parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e93fae63-34a6-4939-9360-cc7dd287b2cf_characterizeImage_1101466_62.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e93fae63-34a6-4939-9360-cc7dd287b2cf_characterizeImage_1101466_62.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/f28c1b42-2f3a-4ce4-b237-514b12a7f5b3_isr_1101466_62.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/f28c1b42-2f3a-4ce4-b237-514b12a7f5b3_isr_1101466_62.stderr parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44b5ebe4c2cebc2fd3d9b1f945aed234 with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash b038e83ff8b63c41d95539b582be287f parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e0e94eca-c0f5-49ce-983b-aece7e9b17a9_characterizeImage_1101475_7.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e0e94eca-c0f5-49ce-983b-aece7e9b17a9_characterizeImage_1101475_7.stderr parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c69aface-bd6c-4d63-a5b0-87771cbebbed_isr_1101475_7.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/c69aface-bd6c-4d63-a5b0-87771cbebbed_isr_1101475_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:13 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6c3b670011bbeb62009fedf66ddb0cd with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 473c22008597eeb5bae037b1caecce58 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b77f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c968575d-6bec-40ec-8c9c-41316090f190_characterizeImage_1101466_48.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c968575d-6bec-40ec-8c9c-41316090f190_characterizeImage_1101466_48.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/08459fe1-d266-4b84-ad60-3e688a5b60bf_isr_1101466_48.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101466/08459fe1-d266-4b84-ad60-3e688a5b60bf_isr_1101466_48.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f730a70cdc1bb9c08e368fe157f3eab0 with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 0b19f5b7b5c65f5561cd4bf0ae531cfa parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b75300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/4ee82821-88e4-4398-adc4-a96d4b9dad39_characterizeImage_1101475_37.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/4ee82821-88e4-4398-adc4-a96d4b9dad39_characterizeImage_1101475_37.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/6d3c01ba-a09e-42e3-bc63-54467ba64581_isr_1101475_37.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101475/6d3c01ba-a09e-42e3-bc63-54467ba64581_isr_1101475_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:13 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 706986fb37c34165ea070be6a82d624e with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 78ec10bcd83b09837056e8a72bf613a6 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b77e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/f83e75c4-ed4e-4297-a10e-3fe154830593_characterizeImage_1101469_52.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/f83e75c4-ed4e-4297-a10e-3fe154830593_characterizeImage_1101469_52.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/c83b8df3-dafa-4bd7-b78e-025493131df2_isr_1101469_52.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/isr/1101469/c83b8df3-dafa-4bd7-b78e-025493131df2_isr_1101469_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:14 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:14 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:15 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:15 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:15 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:16 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:16 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:16 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:17 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:17 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:17 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:18 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:18 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 25 try 0 failed parsl.dataflow.dflow INFO: Task 25 marked for retry parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/df2478d3-b391-4d93-a2cd-a87ad591e22a_characterizeImage_1101466_61.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/df2478d3-b391-4d93-a2cd-a87ad591e22a_characterizeImage_1101466_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash aeb0ceee47fe18cbd564a4fc3350daf1 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620776a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 1 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/df2478d3-b391-4d93-a2cd-a87ad591e22a_characterizeImage_1101466_61.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/df2478d3-b391-4d93-a2cd-a87ad591e22a_characterizeImage_1101466_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:19 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:20 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:21 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:22 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:22 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:23 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:23 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 145 try 0 failed parsl.dataflow.dflow INFO: Task 145 marked for retry parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7c79a3c6-e56d-4b40-bd06-8eb914dec934_characterizeImage_1101466_2.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7c79a3c6-e56d-4b40-bd06-8eb914dec934_characterizeImage_1101466_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 33baa97308adde9ae27015e5231efd7d parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620be7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 1 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7c79a3c6-e56d-4b40-bd06-8eb914dec934_characterizeImage_1101466_2.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7c79a3c6-e56d-4b40-bd06-8eb914dec934_characterizeImage_1101466_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:24 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:26 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:28 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 PD 17226237 PD 17226238 PD 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:29 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 PD 17226237 PD 17226238 PD 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:32 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 PD 17226237 PD 17226238 PD 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:33 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:35 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:37 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 81 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:03:41 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:03:41 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:43 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:44 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:46 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:48 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 979c4f9f3a8f942fa94caa3ee195972e with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 6c7032a2c496d24b3dc7938151ddf4b9 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5ec2190b-d156-4638-9fc3-605182f7842e_calibrate_1101466_18.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5ec2190b-d156-4638-9fc3-605182f7842e_calibrate_1101466_18.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/5e5a1037-ea0a-4a57-a366-dd5b3a210805_characterizeImage_1101466_18.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/5e5a1037-ea0a-4a57-a366-dd5b3a210805_characterizeImage_1101466_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:49 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:51 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:53 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fc5468478733b35412528a1c76389c0 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 708af50befa8ba787158e23c37d2ab5b parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/26d87726-3962-4b78-acb3-b8690c3413ab_calibrate_1101466_30.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/26d87726-3962-4b78-acb3-b8690c3413ab_calibrate_1101466_30.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/6ce55a8b-1d2b-4d52-a7a0-707eadb6a4a2_characterizeImage_1101466_30.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/6ce55a8b-1d2b-4d52-a7a0-707eadb6a4a2_characterizeImage_1101466_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:54 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da98c6f723e75b6f313d8dd6b47fad4b with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 02562d0ab739e693af35b8c06a1eddf7 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162077f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/7accc0e7-bbe9-48a9-bf7e-794b3cb34e28_calibrate_1101472_31.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/7accc0e7-bbe9-48a9-bf7e-794b3cb34e28_calibrate_1101472_31.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/c75f1f98-2b9d-4b52-b8ee-62a17824ca1e_characterizeImage_1101472_31.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/c75f1f98-2b9d-4b52-b8ee-62a17824ca1e_characterizeImage_1101472_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:56 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 427249a4c400f17c4726b35a76e8ea8d with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 79a2c4027ee1e783bd10fdb1463544b3 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1d94e71e-de00-41cb-af8b-8e33c2673260_calibrate_1101466_1.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1d94e71e-de00-41cb-af8b-8e33c2673260_calibrate_1101466_1.stderr parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/fb9705a0-5310-483c-86cd-b9a986e383fc_characterizeImage_1101466_1.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/fb9705a0-5310-483c-86cd-b9a986e383fc_characterizeImage_1101466_1.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9dde1540f513bc0feb4dd4ed33c1648 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 6bed7ad28daa74ee3280e3a63e965f98 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162076520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/3e1a0a67-edde-47aa-8643-168b1e9525a5_calibrate_1101469_50.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/3e1a0a67-edde-47aa-8643-168b1e9525a5_calibrate_1101469_50.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/44f91728-b4fa-41dd-8108-f7b1c5d6c648_characterizeImage_1101469_50.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/44f91728-b4fa-41dd-8108-f7b1c5d6c648_characterizeImage_1101469_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:03:58 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8a99589813d545632b2476be57afbab with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash aa0f23c17fd5ccd996125f9aff962ba1 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620be8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/3da22e3a-763b-4943-9d2c-274d04c8e38e_calibrate_1101469_40.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/3da22e3a-763b-4943-9d2c-274d04c8e38e_calibrate_1101469_40.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/fe77936e-b7d0-4172-b713-0bca6f0eb458_characterizeImage_1101469_40.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/fe77936e-b7d0-4172-b713-0bca6f0eb458_characterizeImage_1101469_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:00 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:02 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e21ccc325016de57f0c174e0b0b17436 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash f9f9e39a58b59764fb4f2b4e5591cd9a parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7a147a2d-7203-4bf9-9619-8f2d405dad09_calibrate_1101469_47.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7a147a2d-7203-4bf9-9619-8f2d405dad09_calibrate_1101469_47.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/fe470721-96d0-4871-beca-c0bdcb620ed3_characterizeImage_1101469_47.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/fe470721-96d0-4871-beca-c0bdcb620ed3_characterizeImage_1101469_47.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1844612fa0f45868e57f3926fd0f6e09 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash c0ba5f4490eb3175ca40d45d065835d7 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/97cc0b8e-97e6-4128-ba0d-9ce7dfd751a0_calibrate_1101466_34.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/97cc0b8e-97e6-4128-ba0d-9ce7dfd751a0_calibrate_1101466_34.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/03d23cc2-5b22-41ea-93f9-478e668a0c82_characterizeImage_1101466_34.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/03d23cc2-5b22-41ea-93f9-478e668a0c82_characterizeImage_1101466_34.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb1fc970921beeca5240b115229f1285 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 2d62f8fca5b658d53ff25a15ce1880a6 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/90ca7623-bc21-495e-8839-c3f8ac7d2157_calibrate_1101469_32.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/90ca7623-bc21-495e-8839-c3f8ac7d2157_calibrate_1101469_32.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/83504901-9919-4c78-a8ef-2a7bad58d3a2_characterizeImage_1101469_32.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/83504901-9919-4c78-a8ef-2a7bad58d3a2_characterizeImage_1101469_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:06 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry becf0c8764607f7664bfcc5807509053 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash bffce22ad8b8a3954476d3273477dcb8 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/aa613804-4eb1-4b49-ab6a-314e5c855974_calibrate_1101469_14.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/aa613804-4eb1-4b49-ab6a-314e5c855974_calibrate_1101469_14.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6da3c72f-a0b6-4efc-b775-9d656f1d8fd4_characterizeImage_1101469_14.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6da3c72f-a0b6-4efc-b775-9d656f1d8fd4_characterizeImage_1101469_14.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8be44b864314cd910ff46986e7bd7ad4 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 0960b7d79e7f1a338f9b2816e2ce055c parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162075260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1cd731e0-e740-4787-b090-47d96fcd2b16_calibrate_1101466_14.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1cd731e0-e740-4787-b090-47d96fcd2b16_calibrate_1101466_14.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/878905c1-cddc-457a-acc4-ba31697cb8d2_characterizeImage_1101466_14.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/878905c1-cddc-457a-acc4-ba31697cb8d2_characterizeImage_1101466_14.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c0133d460e6aadd1d5cec2cde8cd27e with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 2ed8cc3039c07336c37fd67c15a08774 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162075080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/479ea0e9-a820-4aa8-b92f-78b0c3207c05_calibrate_1101472_40.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/479ea0e9-a820-4aa8-b92f-78b0c3207c05_calibrate_1101472_40.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/07bb2f70-306d-4098-a07f-cf2a707754fa_characterizeImage_1101472_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/07bb2f70-306d-4098-a07f-cf2a707754fa_characterizeImage_1101472_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:09 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcbe5d910383353ec530f28573611689 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 7611e6edfccceec6afcc557e3bf5d4be parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162076980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/021c9f26-f89f-4a0d-8d3f-6464855e1737_calibrate_1101469_38.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/021c9f26-f89f-4a0d-8d3f-6464855e1737_calibrate_1101469_38.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/d07bef3f-7f27-41e1-9b8e-fd1f9f4d04f8_characterizeImage_1101469_38.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/d07bef3f-7f27-41e1-9b8e-fd1f9f4d04f8_characterizeImage_1101469_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:10 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:12 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry faa0d302f381facc97ed5e9099d100a0 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 1765fd105dec064b98020b7d7c7622e1 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162076840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ba186166-dd24-4b03-8db5-b0256c4a9007_calibrate_1101472_49.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ba186166-dd24-4b03-8db5-b0256c4a9007_calibrate_1101472_49.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4405056c-6f80-4de3-aaeb-9aedc1bb05c8_characterizeImage_1101472_49.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4405056c-6f80-4de3-aaeb-9aedc1bb05c8_characterizeImage_1101472_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:14 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19ef10963e6e7a9508c51bde2aae1925 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 4a2232ff55a97e10018e96dbcf65e2ba parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2cddefd4-5bfb-43b1-8888-34958b2ea148_calibrate_1101469_57.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2cddefd4-5bfb-43b1-8888-34958b2ea148_calibrate_1101469_57.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/8506dfee-d3a6-44c8-b2ec-998082e449ca_characterizeImage_1101469_57.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/8506dfee-d3a6-44c8-b2ec-998082e449ca_characterizeImage_1101469_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:15 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dabe638fe72d9416f5692fbd099a821 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash b8f27b0f6d2ecd3e091fcd4ec4836074 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/8df4d422-3b3b-4687-b894-86d067d6d9b8_calibrate_1101475_24.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/8df4d422-3b3b-4687-b894-86d067d6d9b8_calibrate_1101475_24.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/13fc318d-4289-449e-9140-ef1088f20a73_characterizeImage_1101475_24.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/13fc318d-4289-449e-9140-ef1088f20a73_characterizeImage_1101475_24.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5ff96be9856ecdf4c1f4603900c0378 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash f36c4e0efcdee72a1457bfb90b65c6a5 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620987c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/c6f4de20-b76c-41d3-83de-e2c267652969_calibrate_1101472_18.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/c6f4de20-b76c-41d3-83de-e2c267652969_calibrate_1101472_18.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/06a9bae1-ec87-484c-8cda-4112c1d7440b_characterizeImage_1101472_18.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/06a9bae1-ec87-484c-8cda-4112c1d7440b_characterizeImage_1101472_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:17 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 054663d5027a8b0d1c9c61bd3d1804ee with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 8e5d31579c9d77ee4ad4b08b6365104c parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/447aaaf6-245e-4a26-b11c-4eb033b834f8_calibrate_1101469_15.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/447aaaf6-245e-4a26-b11c-4eb033b834f8_calibrate_1101469_15.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2387772d-c1fe-49d7-acf7-f237679602a3_characterizeImage_1101469_15.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2387772d-c1fe-49d7-acf7-f237679602a3_characterizeImage_1101469_15.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0b38e236aedc352123bfcc09b7627e1 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash f341860d28ea4720cc15e28911f4678c parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/bb9f3b50-eb8d-4528-9445-0334688b9f87_calibrate_1101475_31.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/bb9f3b50-eb8d-4528-9445-0334688b9f87_calibrate_1101475_31.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8757ec0b-e726-4075-82fe-d119568bea5d_characterizeImage_1101475_31.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8757ec0b-e726-4075-82fe-d119568bea5d_characterizeImage_1101475_31.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c24cb5fb4d80bce8a462e502ae0481af with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 017f77177806ceb7d453c5700c1a93f1 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620984a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ecc09f7b-7e34-44b5-8e22-5dd8f23dde49_calibrate_1101472_24.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ecc09f7b-7e34-44b5-8e22-5dd8f23dde49_calibrate_1101472_24.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/74d61190-e67b-43bc-8c23-7b897d94851e_characterizeImage_1101472_24.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/74d61190-e67b-43bc-8c23-7b897d94851e_characterizeImage_1101472_24.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 813db91eab3fa515478e2b8dd34b0bc3 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash f4774ca7028248e41f8b22049482f2ff parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162077560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/881f5d21-c888-4f88-a548-927b6c8ef6b3_calibrate_1101469_5.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/881f5d21-c888-4f88-a548-927b6c8ef6b3_calibrate_1101469_5.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/270bf27b-6e1f-4d52-8279-486140c6c68c_characterizeImage_1101469_5.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/270bf27b-6e1f-4d52-8279-486140c6c68c_characterizeImage_1101469_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.app.errors DEBUG: Reraising exception of type parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:19 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow DEBUG: Task 295 try 0 failed parsl.dataflow.dflow INFO: Task 295 marked for retry parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2500a4d4-f34a-47e2-a1f8-8543d55c832d_characterizeImage_1101469_2.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2500a4d4-f34a-47e2-a1f8-8543d55c832d_characterizeImage_1101469_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash ff0c04f45cf2d2704887381b773a02b2 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b044a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 1 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2500a4d4-f34a-47e2-a1f8-8543d55c832d_characterizeImage_1101469_2.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2500a4d4-f34a-47e2-a1f8-8543d55c832d_characterizeImage_1101469_2.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f756bffec60563ac1104b3f5b00160e with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash ce2835d76eb3f00893a1d05499957233 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/82f820ea-31f3-4252-a6ed-8bc035e95e69_calibrate_1101466_27.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/82f820ea-31f3-4252-a6ed-8bc035e95e69_calibrate_1101466_27.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/8c9199be-a8df-4956-b55c-79739cbc71a1_characterizeImage_1101466_27.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/8c9199be-a8df-4956-b55c-79739cbc71a1_characterizeImage_1101466_27.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df5aeee5d0ab3abf9aa4948b5c34fce3 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 121ec43af6f8afae07c15aa677eecbad parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/21128926-d2f1-422c-b3d8-d4eabdc7300e_calibrate_1101466_57.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/21128926-d2f1-422c-b3d8-d4eabdc7300e_calibrate_1101466_57.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/975abd01-2068-44d3-b9a1-1c1ea617f4af_characterizeImage_1101466_57.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/975abd01-2068-44d3-b9a1-1c1ea617f4af_characterizeImage_1101466_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:20 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:22 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23c26ccd19a22f85d38cae24fcabc2af with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash a8ced5c98d673d63aeee8d8ace3bb4fc parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162076f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4e217ba9-8757-4008-81d2-fa5031576dd5_calibrate_1101466_60.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4e217ba9-8757-4008-81d2-fa5031576dd5_calibrate_1101466_60.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c5608d05-ccaf-41e1-b310-21842549b308_characterizeImage_1101466_60.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c5608d05-ccaf-41e1-b310-21842549b308_characterizeImage_1101466_60.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27c25db796e8fcde96b4633bb280cfed with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 1002281313920bbc7daaebc38573077a parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/e947d18d-261c-4520-b663-e01e9cf1c742_calibrate_1101469_27.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/e947d18d-261c-4520-b663-e01e9cf1c742_calibrate_1101469_27.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/876d0552-45dc-4c09-84eb-60fed53ae104_characterizeImage_1101469_27.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/876d0552-45dc-4c09-84eb-60fed53ae104_characterizeImage_1101469_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:23 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9438f0911fb1f4e6fc2b47e274e8a358 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash fd2fe7679a8ef412355d5382db617f82 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b70b3a48-e067-485d-a8d4-70537f635d82_calibrate_1101466_12.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b70b3a48-e067-485d-a8d4-70537f635d82_calibrate_1101466_12.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/ee7b1ab5-87eb-4397-82d7-1e3232b99f6e_characterizeImage_1101466_12.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/ee7b1ab5-87eb-4397-82d7-1e3232b99f6e_characterizeImage_1101466_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:24 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b5cc6dd0c3297316f6268c56e31e6fb with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 4f5ed16b9001d34b5561a0622bb3b22c parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/cddbe314-394e-4a51-8051-976d1a7d2398_calibrate_1101469_7.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/cddbe314-394e-4a51-8051-976d1a7d2398_calibrate_1101469_7.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/16483b90-c8c3-4295-9489-bc2036c87424_characterizeImage_1101469_7.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/16483b90-c8c3-4295-9489-bc2036c87424_characterizeImage_1101469_7.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b60cd0769a3db0dd224a7fea872d8bc1 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 95effa69c0ff7487bfaf90f5fd6980d9 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/f289ca8d-2db7-416a-82d9-5f1d3ce804ba_calibrate_1101475_27.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/f289ca8d-2db7-416a-82d9-5f1d3ce804ba_calibrate_1101475_27.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8e5ffebb-f698-426e-a0db-cbbb09680f62_characterizeImage_1101475_27.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8e5ffebb-f698-426e-a0db-cbbb09680f62_characterizeImage_1101475_27.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eeb684ba0fb25c4f3a1146311cc81db4 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 3e6abd2ce6f1643786b5a98f40bb3c1e parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/ee4a175f-eb03-40e4-aec1-4682b38f327a_calibrate_1101466_6.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/ee4a175f-eb03-40e4-aec1-4682b38f327a_calibrate_1101466_6.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/4dc37ee3-8c99-4447-8de9-f145e33f34cf_characterizeImage_1101466_6.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/4dc37ee3-8c99-4447-8de9-f145e33f34cf_characterizeImage_1101466_6.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41578f39ade8c5259fb14462ebffe85c with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 00cf3b86f68d6b83847ec06074ba610f parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/71a11247-94cb-4b1b-adae-e20dbadde565_calibrate_1101469_51.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/71a11247-94cb-4b1b-adae-e20dbadde565_calibrate_1101469_51.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c889e4bc-5b76-4ec9-b925-6d539b2f8c5e_characterizeImage_1101469_51.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c889e4bc-5b76-4ec9-b925-6d539b2f8c5e_characterizeImage_1101469_51.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bc292dc1d74538277dc000271ec7880 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 2b528db9a7b0f29c919c06b9426b4ba1 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162098e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/916ebda2-617e-4417-9041-e43e3e782b9c_calibrate_1101475_35.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/916ebda2-617e-4417-9041-e43e3e782b9c_calibrate_1101475_35.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/75d8076e-de60-4f7d-a4ee-670587b0935b_characterizeImage_1101475_35.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/75d8076e-de60-4f7d-a4ee-670587b0935b_characterizeImage_1101475_35.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 280ef39981d3f4a66949846170e968cb with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash e1897ceccb6701d80f6ca3b60ab34beb parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162075ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/6f4717b6-0f2e-4aca-af75-e87b18909b17_calibrate_1101472_15.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/6f4717b6-0f2e-4aca-af75-e87b18909b17_calibrate_1101472_15.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/79ab60ee-11fa-4715-b284-b6a0d45f3baf_characterizeImage_1101472_15.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/79ab60ee-11fa-4715-b284-b6a0d45f3baf_characterizeImage_1101472_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:26 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4d34ff78331d51e2c7b8a8bae9c78c5 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 581627e1ed9d70f74366ea14b093a772 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162075620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/1a54754e-b0f3-4ddf-b3ca-c04574ebd459_calibrate_1101475_52.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/1a54754e-b0f3-4ddf-b3ca-c04574ebd459_calibrate_1101475_52.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/47839087-0c61-4013-bcc4-86caedbbae62_characterizeImage_1101475_52.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/47839087-0c61-4013-bcc4-86caedbbae62_characterizeImage_1101475_52.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 480e035f32de70d8c3b684c18baf4d91 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 018de0128233e74bffb7812dc4913dc1 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/8f7210c8-4994-4c0d-90e3-1e13e8083692_calibrate_1101472_51.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/8f7210c8-4994-4c0d-90e3-1e13e8083692_calibrate_1101472_51.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d52259b2-abb8-4283-90d5-9004d68e8f70_characterizeImage_1101472_51.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d52259b2-abb8-4283-90d5-9004d68e8f70_characterizeImage_1101472_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:28 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de76548a185a998323eeb88233f3b5c1 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 3a55b27297bfabafc186939301052aff parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/a31416da-5b5b-49a5-8d03-1d845bb1d1b6_calibrate_1101469_8.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/a31416da-5b5b-49a5-8d03-1d845bb1d1b6_calibrate_1101469_8.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/b4593fbc-8d9e-4f25-99f6-e6ce9890435c_characterizeImage_1101469_8.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/b4593fbc-8d9e-4f25-99f6-e6ce9890435c_characterizeImage_1101469_8.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29fadb05ff1f9ebc7c32001419101ef8 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 64cade800efbb20739cb4e3b05bb6a58 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/1de05074-3a3c-42b3-830b-0184fddb113f_calibrate_1101475_51.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/1de05074-3a3c-42b3-830b-0184fddb113f_calibrate_1101475_51.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/d83c6c4d-b95b-450b-85cf-c6b2f73058ba_characterizeImage_1101475_51.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/d83c6c4d-b95b-450b-85cf-c6b2f73058ba_characterizeImage_1101475_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:30 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 538 try 0 failed parsl.dataflow.dflow INFO: Task 538 marked for retry parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/075b50ef-9076-43c9-a6f7-f56866eb0314_characterizeImage_1101475_2.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/075b50ef-9076-43c9-a6f7-f56866eb0314_characterizeImage_1101475_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 77b012388d3e1bae2ab72b37985a85a1 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 1 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/075b50ef-9076-43c9-a6f7-f56866eb0314_characterizeImage_1101475_2.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/075b50ef-9076-43c9-a6f7-f56866eb0314_characterizeImage_1101475_2.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79eb16f3a0ed319adb4b98f3d718c74d with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 55f3b7d9fb274af9a72ba6b0e7377681 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/dc0170ee-77fa-46b8-b950-e8d7b4092574_calibrate_1101469_34.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/dc0170ee-77fa-46b8-b950-e8d7b4092574_calibrate_1101469_34.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/18446002-9c1e-4765-9ed7-464bc2b481ff_characterizeImage_1101469_34.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/18446002-9c1e-4765-9ed7-464bc2b481ff_characterizeImage_1101469_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:32 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 978d5e4ccc000685ed65d9fd101b4e22 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash d0b83639dd6c17fba88ee0cbd39bf566 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ec1356fd-7d8c-4d93-9588-2f240889cd75_calibrate_1101472_39.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ec1356fd-7d8c-4d93-9588-2f240889cd75_calibrate_1101472_39.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/21185a09-1896-4428-b512-4c0f20819bb6_characterizeImage_1101472_39.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/21185a09-1896-4428-b512-4c0f20819bb6_characterizeImage_1101472_39.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05438464615ad439b02f8855b886eab6 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 5c81d08ac8cab9b82495df565a386297 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/4bc6c48c-e631-4b93-ac9f-ae6cbd084f9f_calibrate_1101469_11.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/4bc6c48c-e631-4b93-ac9f-ae6cbd084f9f_calibrate_1101469_11.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6947a0ef-1a11-4641-97e4-01254842c014_characterizeImage_1101469_11.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6947a0ef-1a11-4641-97e4-01254842c014_characterizeImage_1101469_11.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d321b617955d259054355cd98724e370 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 159fa80b5ea390b685f3826db0e87b4e parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/76b0b623-6813-4147-820b-fbf6ed440d0f_calibrate_1101472_52.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/76b0b623-6813-4147-820b-fbf6ed440d0f_calibrate_1101472_52.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/f69798d2-343b-4823-8425-e0edaf34406e_characterizeImage_1101472_52.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/f69798d2-343b-4823-8425-e0edaf34406e_characterizeImage_1101472_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:35 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cab63387aceb0c60b45d5d2651ab0f41 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash a7e48b184bdcaebd9db272b12fff67d1 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ade980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/eed18bc9-60f5-43da-836d-d4323fdd766e_calibrate_1101469_13.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/eed18bc9-60f5-43da-836d-d4323fdd766e_calibrate_1101469_13.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/5b87aab7-3c86-42a5-be9e-a6b0015fe9f0_characterizeImage_1101469_13.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/5b87aab7-3c86-42a5-be9e-a6b0015fe9f0_characterizeImage_1101469_13.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e95e1598bc450e1e1cf2e56062aabb5a with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 1f2fac09bdc25e1b02999583720433a0 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162077240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5fb79185-c4bc-4c80-b04e-be24be4190f1_calibrate_1101475_25.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5fb79185-c4bc-4c80-b04e-be24be4190f1_calibrate_1101475_25.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/eeae8028-2929-47c4-830b-a8554037e194_characterizeImage_1101475_25.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/eeae8028-2929-47c4-830b-a8554037e194_characterizeImage_1101475_25.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5b698a78fd4766083ef000553c3bb82 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 93d847fffa4db9411df6eac2eb907675 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/00d87b46-32a5-4503-9747-6891c03a8e03_calibrate_1101466_16.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/00d87b46-32a5-4503-9747-6891c03a8e03_calibrate_1101466_16.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/5421573e-2683-49f9-b486-587fb19b8e57_characterizeImage_1101466_16.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/5421573e-2683-49f9-b486-587fb19b8e57_characterizeImage_1101466_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:37 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f1e47458be3d847c4c1aaf30be56c38 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash dbb74087e8505fe868a7c9f8c15fd9ae parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620758a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/0c92520d-031b-4078-bcf0-addf3fdcef00_calibrate_1101475_13.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/0c92520d-031b-4078-bcf0-addf3fdcef00_calibrate_1101475_13.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/ff1d0d4c-210c-476d-b738-00ef4c000a0a_characterizeImage_1101475_13.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/ff1d0d4c-210c-476d-b738-00ef4c000a0a_characterizeImage_1101475_13.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4da74f386cd389959428d49819118ce with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash f475a85bfae030f99522aa9a8379a3ca parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d9bdab03-ad31-407f-bcca-cee8145c8226_calibrate_1101472_58.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d9bdab03-ad31-407f-bcca-cee8145c8226_calibrate_1101472_58.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/92d7bbb6-900e-4f50-a39b-bd96beb90e3c_characterizeImage_1101472_58.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/92d7bbb6-900e-4f50-a39b-bd96beb90e3c_characterizeImage_1101472_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:38 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:40 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecff2e788a69b69ee8e788a6fa53b30e with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash bda879fa5ccbaa1456386304ba56a419 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/a7ffd712-e288-4b5f-a753-aa64815cdd60_calibrate_1101469_20.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/a7ffd712-e288-4b5f-a753-aa64815cdd60_calibrate_1101469_20.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/8231f787-ff03-4b71-9dbc-52f8a7b5d375_characterizeImage_1101469_20.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/8231f787-ff03-4b71-9dbc-52f8a7b5d375_characterizeImage_1101469_20.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c5cd686589527ca3ac25e04e8a22146 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 2c71ec6ce35c2090268813f7378d25df parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/41664c2a-5762-4d5d-9a6f-fbfb806431eb_calibrate_1101469_33.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/41664c2a-5762-4d5d-9a6f-fbfb806431eb_calibrate_1101469_33.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c03015fc-6115-4583-8b5c-12702bb13f94_characterizeImage_1101469_33.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c03015fc-6115-4583-8b5c-12702bb13f94_characterizeImage_1101469_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:42 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:43 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:45 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05fae351987ce0861f663c7cf71c6f8d with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash d5262e615dbe417d06fe1ff8f128eeaf parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5fe4f49d-b466-4123-ae71-0c900fe94437_calibrate_1101475_54.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5fe4f49d-b466-4123-ae71-0c900fe94437_calibrate_1101475_54.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/cf2318d1-684f-4227-8d9b-7d76a32a75e1_characterizeImage_1101475_54.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/cf2318d1-684f-4227-8d9b-7d76a32a75e1_characterizeImage_1101475_54.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 479f17beec169e9ce2d1933bcbd7e3bb with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash e869800397392951c1c4ac357527eacf parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/e591665f-8b01-4ed2-969f-10b3408241d6_calibrate_1101475_53.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/e591665f-8b01-4ed2-969f-10b3408241d6_calibrate_1101475_53.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/058837e1-73bd-4b61-8346-d824e4454371_characterizeImage_1101475_53.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/058837e1-73bd-4b61-8346-d824e4454371_characterizeImage_1101475_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:46 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1dc3d31dc0dbc873244381fbea98a10 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 721d3896c451e1fc6a9288cbd1b39bff parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c3f40ae4-763d-4e4e-976b-8a046de45702_calibrate_1101475_5.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c3f40ae4-763d-4e4e-976b-8a046de45702_calibrate_1101475_5.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/5b7627c3-83ca-4602-8749-6afe8f249e28_characterizeImage_1101475_5.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/5b7627c3-83ca-4602-8749-6afe8f249e28_characterizeImage_1101475_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:48 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b4d65aa96fae38779d83417f6e49f18 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash af125c4fd85b0e073329914993e37af2 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1873e2c7-6e55-4284-8ace-6620d2d44163_calibrate_1101466_17.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1873e2c7-6e55-4284-8ace-6620d2d44163_calibrate_1101466_17.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/106bdb6a-b607-42f1-a27e-2e26a82a558a_characterizeImage_1101466_17.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/106bdb6a-b607-42f1-a27e-2e26a82a558a_characterizeImage_1101466_17.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef291b36a9fed9db10a71f90c0245ab6 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 8dfa8ad5353694a06454675ecbf40b5d parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ade340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/2419fb2a-dfd4-41d2-ac70-ee5f346ba3de_calibrate_1101475_9.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/2419fb2a-dfd4-41d2-ac70-ee5f346ba3de_calibrate_1101475_9.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8def3e0e-f98e-40f5-aa5a-1d9ba6035f09_characterizeImage_1101475_9.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8def3e0e-f98e-40f5-aa5a-1d9ba6035f09_characterizeImage_1101475_9.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fef8ef9b2a1894c1679941c169a69a3 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 6ec54367e7a5de71dcaaa5800f0d9102 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162098ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c3854330-45d4-4856-9177-945e01b6541e_calibrate_1101475_60.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c3854330-45d4-4856-9177-945e01b6541e_calibrate_1101475_60.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/4bf596b0-0d06-4a05-8e7b-93753a267111_characterizeImage_1101475_60.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/4bf596b0-0d06-4a05-8e7b-93753a267111_characterizeImage_1101475_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:50 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed6116c934d9c12eb43bfe40298f86e9 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash f8ae7a353fd7742d246bdccaf8cdef76 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/197c69b5-2919-42e6-bd7e-7943879e10f8_calibrate_1101472_1.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/197c69b5-2919-42e6-bd7e-7943879e10f8_calibrate_1101472_1.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/1c80089c-af42-4a1f-a023-b95b5b5d5422_characterizeImage_1101472_1.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/1c80089c-af42-4a1f-a023-b95b5b5d5422_characterizeImage_1101472_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:51 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9faacc789c1423c88558d08a4f950e18 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 1effb4a31340c7009b541b05eb12bfbb parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160add9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/2bc10262-bb8c-41f0-942a-75349a5a449f_calibrate_1101472_56.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/2bc10262-bb8c-41f0-942a-75349a5a449f_calibrate_1101472_56.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8652ad9b-12d4-45f8-a2e9-4495c11b7e41_characterizeImage_1101472_56.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8652ad9b-12d4-45f8-a2e9-4495c11b7e41_characterizeImage_1101472_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:53 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d69ad1815f761cb00a2ed864f17513a with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 972f5826e5addd0e37bbf1d51b2c4122 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/efcb7b7c-ecb7-4a74-9220-8357ae85d9cd_calibrate_1101475_50.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/efcb7b7c-ecb7-4a74-9220-8357ae85d9cd_calibrate_1101475_50.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/ae7bbbdd-dd85-416d-a437-236ed3689694_characterizeImage_1101475_50.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/ae7bbbdd-dd85-416d-a437-236ed3689694_characterizeImage_1101475_50.stderr parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 711066079e7519628a67da3d67129c6e with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash d6eab96d9128e379bdccad341e4637c9 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/85639a6f-31d4-4058-bfc6-169078579509_calibrate_1101466_19.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/85639a6f-31d4-4058-bfc6-169078579509_calibrate_1101466_19.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c9ed3001-7e27-43a3-be41-5ba859b848c3_characterizeImage_1101466_19.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c9ed3001-7e27-43a3-be41-5ba859b848c3_characterizeImage_1101466_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:54 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b4700b9636b2e719df10a8118904e93 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 6a0ee33fd955291354f65ff54f337770 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/a8d9c19c-5170-4c09-9615-055d4024c64e_calibrate_1101475_41.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/a8d9c19c-5170-4c09-9615-055d4024c64e_calibrate_1101475_41.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e2fb4f39-4f4f-41dd-a3e3-5f8170318bee_characterizeImage_1101475_41.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e2fb4f39-4f4f-41dd-a3e3-5f8170318bee_characterizeImage_1101475_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:56 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37bddcb0d637d76ab4698989300ccb49 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 39bd489d88036ae682f4f663e3805552 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d3831056-42bc-436d-86cb-a18e31973495_calibrate_1101475_14.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d3831056-42bc-436d-86cb-a18e31973495_calibrate_1101475_14.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/dd1eccc9-e2cc-44ae-9e6a-98954b4d4fef_characterizeImage_1101475_14.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/dd1eccc9-e2cc-44ae-9e6a-98954b4d4fef_characterizeImage_1101475_14.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49a6c3458d5e4acdc4070c697518e8bd with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 7de96802104252cf4bfc9eb6a8dc89cf parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/eefeb1ca-1510-45bb-8b65-eba87f3bf573_calibrate_1101472_41.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/eefeb1ca-1510-45bb-8b65-eba87f3bf573_calibrate_1101472_41.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/7e96e7c6-f527-4608-a2bb-6e56b3ccf724_characterizeImage_1101472_41.stdout parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/7e96e7c6-f527-4608-a2bb-6e56b3ccf724_characterizeImage_1101472_41.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:58 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7888dbafbe62a45e1dabf25698d5aff0 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash f6c03e84427373321f2b36de88001192 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162098180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c7155b9e-b97b-4a03-8fca-8333ece6d872_calibrate_1101475_45.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c7155b9e-b97b-4a03-8fca-8333ece6d872_calibrate_1101475_45.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/6d17a6a4-82d4-401f-8720-8b69d9b8aca7_characterizeImage_1101475_45.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/6d17a6a4-82d4-401f-8720-8b69d9b8aca7_characterizeImage_1101475_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:04:59 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea93c988c2bd14222d2300bcf6dfb4d4 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash d38c7ff58ac488bf66f6a5442da0b99b parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160add080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/736b3a5d-61b9-4079-aa31-2696ea40850e_calibrate_1101472_43.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/736b3a5d-61b9-4079-aa31-2696ea40850e_calibrate_1101472_43.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/30890ee0-a024-44c7-83a1-55699578a46b_characterizeImage_1101472_43.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/30890ee0-a024-44c7-83a1-55699578a46b_characterizeImage_1101472_43.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae30306732ca831676d7810e548ac6d2 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash bc4475f38d723b3f7d28a6e514ab0b98 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160add3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/792aa945-e8f5-4959-ab74-87e2dbadbc16_calibrate_1101472_4.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/792aa945-e8f5-4959-ab74-87e2dbadbc16_calibrate_1101472_4.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/10668301-2f00-446c-bf78-d0b40ec4ffe5_characterizeImage_1101472_4.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/10668301-2f00-446c-bf78-d0b40ec4ffe5_characterizeImage_1101472_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:02 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46caecc579c564289e6955f45b766d0f with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 4d94c4924975c875fb7b6ba0aacc204f parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/9c374b7c-dd53-4b39-b199-0e1a91d6610a_calibrate_1101475_48.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/9c374b7c-dd53-4b39-b199-0e1a91d6610a_calibrate_1101475_48.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/20232d0a-58ba-48fc-aa54-b533f4315330_characterizeImage_1101475_48.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/20232d0a-58ba-48fc-aa54-b533f4315330_characterizeImage_1101475_48.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33b3a381b5b144868b384fc45ee280b4 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash d2f51de0d3e9e4fd6310e03c318650c8 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/485fc6a3-2c53-493a-8a5b-d99838ac5d32_calibrate_1101472_6.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/485fc6a3-2c53-493a-8a5b-d99838ac5d32_calibrate_1101472_6.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d8a9d7e1-fa96-4fb2-8dab-79e731b9b976_characterizeImage_1101472_6.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d8a9d7e1-fa96-4fb2-8dab-79e731b9b976_characterizeImage_1101472_6.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97b71c68b81118f348a9e6eb10f8262d with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash db243992e18064bfde3e53d50efed12a parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/79586ed5-1071-4013-ad6d-c0e37e7d088f_calibrate_1101475_44.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/79586ed5-1071-4013-ad6d-c0e37e7d088f_calibrate_1101475_44.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e5efe47c-acec-4352-b499-eb5d44cd9087_characterizeImage_1101475_44.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e5efe47c-acec-4352-b499-eb5d44cd9087_characterizeImage_1101475_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:03 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b2126dba4926f2ecd318c8f35429c9f with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash aa9c9909d5d6d705653c9313d8a80447 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620be2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/d971fe86-0040-4aea-9524-ba3bc3d02b58_calibrate_1101469_37.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/d971fe86-0040-4aea-9524-ba3bc3d02b58_calibrate_1101469_37.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/80e190d1-9cec-4438-b6e3-83a29316ab21_characterizeImage_1101469_37.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/80e190d1-9cec-4438-b6e3-83a29316ab21_characterizeImage_1101469_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:06 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbf969a86967f6f7914750b0b8a1b84d with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash efaf01a3796f47cf0adeabc941f3ffc6 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/e17d54ea-bbf5-431f-923f-c3c9382b8687_calibrate_1101469_12.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/e17d54ea-bbf5-431f-923f-c3c9382b8687_calibrate_1101469_12.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/5fec3049-e658-48be-b1cb-0a077d5afb4f_characterizeImage_1101469_12.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/5fec3049-e658-48be-b1cb-0a077d5afb4f_characterizeImage_1101469_12.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af01352cffa68d0d9f4a5ed7135648bd with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 9403a387ea6ebfaa6a090dc2b4bd51b9 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/2b35b7d5-602d-4311-bbe3-a4d251a2ab2c_calibrate_1101475_29.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/2b35b7d5-602d-4311-bbe3-a4d251a2ab2c_calibrate_1101475_29.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/df28707d-4d0b-4a9e-8ea7-323fda868854_characterizeImage_1101475_29.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/df28707d-4d0b-4a9e-8ea7-323fda868854_characterizeImage_1101475_29.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3a1624636349fa458fa37656a62d449 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 3872ff72d6ea928881f481243c5b83e8 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160add6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/f4902252-cb6f-4fef-b2a1-cf421ed60551_calibrate_1101475_28.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/f4902252-cb6f-4fef-b2a1-cf421ed60551_calibrate_1101475_28.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/76461483-7784-4577-9c5f-8a5113a61b7b_characterizeImage_1101475_28.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/76461483-7784-4577-9c5f-8a5113a61b7b_characterizeImage_1101475_28.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 508ef34dd0a72cfadb490f7347499c29 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 520edbe152bce2278a6f0033f6fa69d1 parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/db199cff-394e-4540-9a44-96ecfa62df1d_calibrate_1101469_19.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/db199cff-394e-4540-9a44-96ecfa62df1d_calibrate_1101469_19.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/8227cfc3-2b46-4448-aad0-9b56a7d4a56d_characterizeImage_1101469_19.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/8227cfc3-2b46-4448-aad0-9b56a7d4a56d_characterizeImage_1101469_19.stderr parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6066f4f0fc789c4aa97157fc7f11bdb4 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 6c209cb61ae8a9a1a216e52203c96f28 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162074cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/05ef43c9-c195-4328-924e-a31d2394e9f0_calibrate_1101469_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/05ef43c9-c195-4328-924e-a31d2394e9f0_calibrate_1101469_41.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/66d7c4d7-4a66-421c-92b9-e0e4621f5e51_characterizeImage_1101469_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/66d7c4d7-4a66-421c-92b9-e0e4621f5e51_characterizeImage_1101469_41.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e57e9b17b15213888c0a380fe6413fc7 with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash d0ced774d9af69a1375172e2ede28751 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b04f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/7d2e7469-7cd1-46ec-a5de-36785db09457_calibrate_1101475_58.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/7d2e7469-7cd1-46ec-a5de-36785db09457_calibrate_1101475_58.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c0b0043b-186a-445c-a3aa-caed089ebdb3_characterizeImage_1101475_58.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c0b0043b-186a-445c-a3aa-caed089ebdb3_characterizeImage_1101475_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:08 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e601194eaf44f60771df1ba2b313ac97 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash d3fe1853091abbad82eba44d04d80572 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b077e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/a58145f2-045a-4f4b-bee6-b13261a59806_calibrate_1101469_43.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/a58145f2-045a-4f4b-bee6-b13261a59806_calibrate_1101469_43.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6f26a06a-0ae4-4dbf-869a-87e29412e118_characterizeImage_1101469_43.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6f26a06a-0ae4-4dbf-869a-87e29412e118_characterizeImage_1101469_43.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ce7307e1a91a892f371d0a4fbf9dc15 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 42a00b70494f4743b546ba6153492d45 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/17176af6-fc87-4c76-9539-9e25a523f718_calibrate_1101472_3.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/17176af6-fc87-4c76-9539-9e25a523f718_calibrate_1101472_3.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/9a4201a7-894f-48b5-8251-556a75b7317b_characterizeImage_1101472_3.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/9a4201a7-894f-48b5-8251-556a75b7317b_characterizeImage_1101472_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:10 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f21f04f376ff51ec88340487e1019d29 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash ba48374ac064c6ec4da8a57b8b0dd91f parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620763e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/49e33f19-90e9-49fe-815e-eb3db7ae72e5_calibrate_1101469_17.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/49e33f19-90e9-49fe-815e-eb3db7ae72e5_calibrate_1101469_17.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/580c5365-e0df-4099-a3d7-7f9ae4ef575b_characterizeImage_1101469_17.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/580c5365-e0df-4099-a3d7-7f9ae4ef575b_characterizeImage_1101469_17.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb7f7373854be1ff8f4e42d424c0f090 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash e934a2acf27eb64c1901a986124eae80 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b39120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/a0db8ced-8329-4109-915c-30f784677d54_calibrate_1101475_49.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/a0db8ced-8329-4109-915c-30f784677d54_calibrate_1101475_49.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/5ccfa9cd-09ae-4ccb-b25c-95e8ae68f975_characterizeImage_1101475_49.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/5ccfa9cd-09ae-4ccb-b25c-95e8ae68f975_characterizeImage_1101475_49.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ffb22e6381a5c69d8c69ba067d10837 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 43f6860c5d782d426258010f03ec38e7 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b387c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/9bbfd03f-f3d1-4d11-84ef-48ee2fbe0fab_calibrate_1101475_57.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/9bbfd03f-f3d1-4d11-84ef-48ee2fbe0fab_calibrate_1101475_57.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/49e04028-2539-4317-a800-2ab88181d789_characterizeImage_1101475_57.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/49e04028-2539-4317-a800-2ab88181d789_characterizeImage_1101475_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:12 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a9ba02dbdbe839ad94f2ebb8674c5a0 with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash f325723523cc9e1b97a47a7dcbdac6f4 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ade020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/ca1a5a30-dd1d-45c8-924d-37998e9e8aac_calibrate_1101466_49.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/ca1a5a30-dd1d-45c8-924d-37998e9e8aac_calibrate_1101466_49.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/70b46cbf-eeed-49f2-9896-8f93c18471c4_characterizeImage_1101466_49.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/70b46cbf-eeed-49f2-9896-8f93c18471c4_characterizeImage_1101466_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:13 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02e9622835aa3eb7d0678a07d731885c with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 3ecbb7eb6b01e43123a0526ff4752e1a parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2e72fed2-43c6-451d-b71a-a7d0590178a6_calibrate_1101466_3.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2e72fed2-43c6-451d-b71a-a7d0590178a6_calibrate_1101466_3.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/318d59b9-3e5b-4ff0-8990-5e60db888b0a_characterizeImage_1101466_3.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/318d59b9-3e5b-4ff0-8990-5e60db888b0a_characterizeImage_1101466_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:15 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f2d9a06e4ff7957b6b18210c114ed74 with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash 0f799ba378cf0fdf6395b3473c5d0475 parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2be8145d-c0eb-453f-99cd-72d6edbe70da_calibrate_1101466_46.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2be8145d-c0eb-453f-99cd-72d6edbe70da_calibrate_1101466_46.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/1e71bb91-a346-4fef-9912-854f6d9c5584_characterizeImage_1101466_46.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/1e71bb91-a346-4fef-9912-854f6d9c5584_characterizeImage_1101466_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:16 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.memoization DEBUG: Storing app cache entry 98d401949c96ea30488a352d447769a5 with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash c2a44bfd974c379c51c89dcf08ac8fdc parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5ea4825b-8eb8-4fea-8b83-3283c34d34e2_calibrate_1101466_52.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5ea4825b-8eb8-4fea-8b83-3283c34d34e2_calibrate_1101466_52.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/4319ca08-584b-4406-bee4-69cae37a5bb4_characterizeImage_1101466_52.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/4319ca08-584b-4406-bee4-69cae37a5bb4_characterizeImage_1101466_52.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc6bf7ff492f7f7155346a240be31deb with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 5a7975e50d56d526bfce6d0bcb6ed3e2 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/f3990caa-7ece-4a3f-964f-81a6754c952d_calibrate_1101469_30.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/f3990caa-7ece-4a3f-964f-81a6754c952d_calibrate_1101469_30.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/ffcd6f5b-c496-4faf-b955-3a1b326fb974_characterizeImage_1101469_30.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/ffcd6f5b-c496-4faf-b955-3a1b326fb974_characterizeImage_1101469_30.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 610 try 0 failed parsl.dataflow.dflow INFO: Task 610 marked for retry parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8f528bae-823c-4669-a83f-423304c44523_characterizeImage_1101472_2.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8f528bae-823c-4669-a83f-423304c44523_characterizeImage_1101472_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 7c59eba3cd2f6e901c8bfee58ac0649a parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 1 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8f528bae-823c-4669-a83f-423304c44523_characterizeImage_1101472_2.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8f528bae-823c-4669-a83f-423304c44523_characterizeImage_1101472_2.stderr parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc20271746db8bb97c12ae6950267080 with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash 55e40ba7a2b8afef4a42cf55b85a040f parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4fff2529-1d44-437a-96e9-a921005cb5a2_calibrate_1101466_35.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4fff2529-1d44-437a-96e9-a921005cb5a2_calibrate_1101466_35.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/98926d4c-aa1c-4e51-8e00-69ec1a43a4fa_characterizeImage_1101466_35.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/98926d4c-aa1c-4e51-8e00-69ec1a43a4fa_characterizeImage_1101466_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:18 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82f60a348f9f7d05acfd0e4a274c8a8b with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash bc1d8f211de6175ab6b8edd28531bb6a parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/c5eed667-a636-468a-9584-8d8fa7639fbc_calibrate_1101466_31.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/c5eed667-a636-468a-9584-8d8fa7639fbc_calibrate_1101466_31.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/79fdcb8d-5720-41e7-bfee-6a4db6a4f7de_characterizeImage_1101466_31.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/79fdcb8d-5720-41e7-bfee-6a4db6a4f7de_characterizeImage_1101466_31.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f61aec43960ae172cf7d0b2a6cd5ebf with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 4f0ae33de9d10bc3b57c921234e1aebb parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b06840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1bbf4df7-aaf3-4d5b-8f58-123bc69fdfc5_calibrate_1101466_50.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1bbf4df7-aaf3-4d5b-8f58-123bc69fdfc5_calibrate_1101466_50.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e459e5e0-4083-463e-a854-f9a1d97a7a8b_characterizeImage_1101466_50.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e459e5e0-4083-463e-a854-f9a1d97a7a8b_characterizeImage_1101466_50.stderr parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 032898f2ac2281f61f4567d8a3f4902b with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 415be7d084f19344618c9e42af617a86 parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ea4c88a5-ee17-4158-8b16-96a9e7f22bdc_calibrate_1101472_19.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ea4c88a5-ee17-4158-8b16-96a9e7f22bdc_calibrate_1101472_19.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/46d5592a-c9fa-427f-9897-221acafea56a_characterizeImage_1101472_19.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/46d5592a-c9fa-427f-9897-221acafea56a_characterizeImage_1101472_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:19 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd5aa1929b87643823370ef1efc99c2a with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 97327e336993c77254724203fad139bf parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b07b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d533ac8a-86bf-436e-9d37-a68ba5108f7e_calibrate_1101472_33.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d533ac8a-86bf-436e-9d37-a68ba5108f7e_calibrate_1101472_33.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/7c0d3bd9-261a-4a4a-89f3-1b325914e223_characterizeImage_1101472_33.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/7c0d3bd9-261a-4a4a-89f3-1b325914e223_characterizeImage_1101472_33.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5187dfd745f73877afb70d5f6afc0743 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 000fa9271587734eb6a08da88736fd08 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b384a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8de0c1ea-a354-477c-aa7c-4a025ad94541_calibrate_1101469_9.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8de0c1ea-a354-477c-aa7c-4a025ad94541_calibrate_1101469_9.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/4ffbbd51-899a-4647-84b2-cbdb640644c5_characterizeImage_1101469_9.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/4ffbbd51-899a-4647-84b2-cbdb640644c5_characterizeImage_1101469_9.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fd242e9d92d4ed6379b1415edeb4a5b with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash b92382bb93b60ad57eb964b718e908d5 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b06200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/48bbc288-7d95-46e0-ae49-dd50e63f2ad3_calibrate_1101469_18.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/48bbc288-7d95-46e0-ae49-dd50e63f2ad3_calibrate_1101469_18.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/972ad8ef-de97-4fef-b14d-7cde4d715215_characterizeImage_1101469_18.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/972ad8ef-de97-4fef-b14d-7cde4d715215_characterizeImage_1101469_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:21 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62ff9d2c404a14d441b38fbd8c0a8dbb with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 588521b41d848657eb3eaf04a2e5b1e6 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b5e1c444-60c0-47bd-9454-05cc387a112a_calibrate_1101466_25.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b5e1c444-60c0-47bd-9454-05cc387a112a_calibrate_1101466_25.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/872e7947-15dc-4a7c-a782-eeee4ce0848f_characterizeImage_1101466_25.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/872e7947-15dc-4a7c-a782-eeee4ce0848f_characterizeImage_1101466_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:22 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f21e48f5535963a804ccb7dcd4a17544 with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash f2faed1ca480ed964a1b7059f5f1c323 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b39760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/8cd3c42f-3623-4d68-9bd5-db4985d7e7a5_calibrate_1101466_42.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/8cd3c42f-3623-4d68-9bd5-db4985d7e7a5_calibrate_1101466_42.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/53b54df1-1b84-4b6b-a2f6-11300bc8d41d_characterizeImage_1101466_42.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/53b54df1-1b84-4b6b-a2f6-11300bc8d41d_characterizeImage_1101466_42.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6a2ab8852f72b2c7d47368f856f3d63 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash b75c4abc1f95e605c61d5acb1871d9f4 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/58309d37-316b-406a-9c59-a1c6e5457ce7_calibrate_1101466_53.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/58309d37-316b-406a-9c59-a1c6e5457ce7_calibrate_1101466_53.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/684b97be-e935-46f3-ab3f-127deae9e818_characterizeImage_1101466_53.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/684b97be-e935-46f3-ab3f-127deae9e818_characterizeImage_1101466_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:24 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01c0785b78d4b08aa92bd61b942f2d4e with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash bc676f4e00d8b47f5984dcfe90ee00b4 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/19abd69c-9725-44d7-91ce-4ee55a2725c8_calibrate_1101469_45.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/19abd69c-9725-44d7-91ce-4ee55a2725c8_calibrate_1101469_45.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/73e9ad96-afed-4f1c-843f-902882e77556_characterizeImage_1101469_45.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/73e9ad96-afed-4f1c-843f-902882e77556_characterizeImage_1101469_45.stderr parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e669fa8ed16a6f437b417a4df9c4d20 with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash ce48b37ccccd60fd102dc5c3c1dfc77e parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b39c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/6cd1a780-a96d-4915-a2e9-dc1d4bfcdff0_calibrate_1101475_47.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/6cd1a780-a96d-4915-a2e9-dc1d4bfcdff0_calibrate_1101475_47.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/6f51b9d1-de6b-4e05-b0d5-b68de7744272_characterizeImage_1101475_47.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/6f51b9d1-de6b-4e05-b0d5-b68de7744272_characterizeImage_1101475_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:25 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c30641b85a49a07ef41bd35e5695c2d with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash b689e645ec2d889a6eb2bef1a319ff39 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d2ffe84e-a8c1-4fbd-8a0e-f91f4f33ee31_calibrate_1101472_54.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d2ffe84e-a8c1-4fbd-8a0e-f91f4f33ee31_calibrate_1101472_54.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/770bcdd2-b4b4-4047-97a3-e7d3ba005835_characterizeImage_1101472_54.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/770bcdd2-b4b4-4047-97a3-e7d3ba005835_characterizeImage_1101472_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:27 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e94c9f83d3aa9165bbc628418df93fd with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 3793b19e305a98f465201dc3ee451e26 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ade660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/99845815-988c-4a0e-a04c-0aedf0f71d24_calibrate_1101469_62.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/99845815-988c-4a0e-a04c-0aedf0f71d24_calibrate_1101469_62.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/e218ca9f-df2e-40bd-a88e-fecca922b35c_characterizeImage_1101469_62.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/e218ca9f-df2e-40bd-a88e-fecca922b35c_characterizeImage_1101469_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:05:29 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:05:29 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2f758a25c51a9ef274fe2eca56c18c3 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash a644360e78192c26721ae0766f05f5f1 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b07f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/d95ecf57-e62c-4d84-a19a-22cbb4fca3f7_calibrate_1101469_58.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/d95ecf57-e62c-4d84-a19a-22cbb4fca3f7_calibrate_1101469_58.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6193ace2-1c64-4586-90b9-b8a47c9ca6be_characterizeImage_1101469_58.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/6193ace2-1c64-4586-90b9-b8a47c9ca6be_characterizeImage_1101469_58.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 592b987316c099da5597f7a01ed455b6 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 84899033fbe631eb29908b8ea3cfbacb parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c6c4fb3f-38d3-45e2-859f-e852fad5ca21_calibrate_1101475_40.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c6c4fb3f-38d3-45e2-859f-e852fad5ca21_calibrate_1101475_40.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/114708e7-153b-436b-9ea0-2427d618c687_characterizeImage_1101475_40.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/114708e7-153b-436b-9ea0-2427d618c687_characterizeImage_1101475_40.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b4734bac2cbc7ae18c5df2493de23ff with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 3f50bf0a83f183ba3aee8ad194eff0f7 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b074c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/190833cf-05d4-4a80-ada2-59c7ad44ccfb_calibrate_1101472_13.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/190833cf-05d4-4a80-ada2-59c7ad44ccfb_calibrate_1101472_13.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8d08afb4-7984-4a76-a1a2-fa879105761c_characterizeImage_1101472_13.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8d08afb4-7984-4a76-a1a2-fa879105761c_characterizeImage_1101472_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:32 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b5e72f670e37b1aebe3adcf48bddec9 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash fb34e5691def00324b4137e18aeb30a5 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b06b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8809b9a9-df83-4e5c-9200-3b808821fa15_calibrate_1101469_46.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8809b9a9-df83-4e5c-9200-3b808821fa15_calibrate_1101469_46.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/3f9311d9-8b19-4bfc-ab6b-fb3bd92b9172_characterizeImage_1101469_46.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/3f9311d9-8b19-4bfc-ab6b-fb3bd92b9172_characterizeImage_1101469_46.stderr parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75596326f1bd4a4eb7ef9d659ad3ade0 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 0777655ae6517db6cba7e3f2c7423954 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609be8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/0e7af567-692e-4a25-8b21-226566fe46f6_calibrate_1101466_20.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/0e7af567-692e-4a25-8b21-226566fe46f6_calibrate_1101466_20.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/08b30b93-aca2-4d1d-b260-2612e5b9f698_characterizeImage_1101466_20.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/08b30b93-aca2-4d1d-b260-2612e5b9f698_characterizeImage_1101466_20.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 982ba0fbc22265f27f0ae9d2fe0add30 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 270bf72aeb2315f9d3d0e506e0b1e20e parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162099760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/8e709f50-2ec0-4cae-8328-7c1990e095f8_calibrate_1101475_16.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/8e709f50-2ec0-4cae-8328-7c1990e095f8_calibrate_1101475_16.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c695400b-ed2b-4bf3-9b12-687215a1c4bd_characterizeImage_1101475_16.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c695400b-ed2b-4bf3-9b12-687215a1c4bd_characterizeImage_1101475_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:34 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3e8d2a718ad666d4d535c41315c88ad with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 3a7786e4799d6fbdcf835569d5c8f630 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/cfcd6480-a75f-40a1-b817-263a486b65ff_calibrate_1101475_18.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/cfcd6480-a75f-40a1-b817-263a486b65ff_calibrate_1101475_18.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/fd117811-7962-4f97-a44e-76f4b7100ff3_characterizeImage_1101475_18.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/fd117811-7962-4f97-a44e-76f4b7100ff3_characterizeImage_1101475_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:36 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68dabeaac15037f5f034e6d199d36eae with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 85929ff12684b82b99b8c437edcca7a1 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160addd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b306772a-de12-4776-b94b-357351edae97_calibrate_1101466_37.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b306772a-de12-4776-b94b-357351edae97_calibrate_1101466_37.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e92c796f-d527-4387-bd56-bd22d013d651_characterizeImage_1101466_37.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e92c796f-d527-4387-bd56-bd22d013d651_characterizeImage_1101466_37.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35c3284ecb5e5d704c30209c623f1328 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 1e3e6deb286724ce85b6313e209c5d43 parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/843dbc43-e951-43ae-9b50-79ab29a08d61_calibrate_1101472_12.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/843dbc43-e951-43ae-9b50-79ab29a08d61_calibrate_1101472_12.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/9c72b743-202d-4360-92a7-c81838929db2_characterizeImage_1101472_12.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/9c72b743-202d-4360-92a7-c81838929db2_characterizeImage_1101472_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:38 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 935c1eeb1f6de917767f8dcdd0420494 with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash 62bab524a5e1ed7a2f8829c7ce583c05 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609be2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/be30fd53-dded-431b-9f80-7e1ab572c6f8_calibrate_1101472_32.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/be30fd53-dded-431b-9f80-7e1ab572c6f8_calibrate_1101472_32.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4e3fe2f6-ba64-4024-b449-e185004e6174_characterizeImage_1101472_32.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4e3fe2f6-ba64-4024-b449-e185004e6174_characterizeImage_1101472_32.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcffdc4cff35794ff73d7f7e19c7e452 with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 9b65097dd38f9a5bcbe078babc2b9383 parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b39bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/587f8cfb-3db2-472d-9471-c9257d75e64b_calibrate_1101469_23.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/587f8cfb-3db2-472d-9471-c9257d75e64b_calibrate_1101469_23.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/155479cd-fc32-44c0-b7a7-591478690722_characterizeImage_1101469_23.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/155479cd-fc32-44c0-b7a7-591478690722_characterizeImage_1101469_23.stderr parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44d29916b610b0be85f499cd47a4a8f5 with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 62096bda1f83a0460bf255854b2b223f parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5cdfd107-c55b-41f6-b246-834dc0a56de6_calibrate_1101475_39.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5cdfd107-c55b-41f6-b246-834dc0a56de6_calibrate_1101475_39.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a3e7e109-3cca-4625-b9ef-cac36a74e2d5_characterizeImage_1101475_39.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a3e7e109-3cca-4625-b9ef-cac36a74e2d5_characterizeImage_1101475_39.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f18b6a8d7ec2d07b650f128d6663fcc4 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 465eaaed77928f8adf69474ebc69dd8a parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b38180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/22f2016b-d900-4f4b-81df-e2672b87a88a_calibrate_1101472_62.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/22f2016b-d900-4f4b-81df-e2672b87a88a_calibrate_1101472_62.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8e25aee7-724d-42be-bccc-01e74dc3c3d6_characterizeImage_1101472_62.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8e25aee7-724d-42be-bccc-01e74dc3c3d6_characterizeImage_1101472_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:41 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f607d9fb656f777f4cf97ef272edae5 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 4d2971bf82189a9c4ba600d1defc1767 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b04900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d9a32bd2-e021-4109-af68-0da6e43dbeca_calibrate_1101472_11.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d9a32bd2-e021-4109-af68-0da6e43dbeca_calibrate_1101472_11.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/cae0bcc7-3725-46df-971b-4878269e0177_characterizeImage_1101472_11.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/cae0bcc7-3725-46df-971b-4878269e0177_characterizeImage_1101472_11.stderr parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4978e3529088660dd51ed471cd0f098b with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash bfdd88aea0cc947f0d6b173c877d58d9 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/478fbb0b-cc53-4dd6-8cc7-87bbbfa1d472_calibrate_1101466_44.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/478fbb0b-cc53-4dd6-8cc7-87bbbfa1d472_calibrate_1101466_44.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/a8825bcc-a28b-40ca-9f20-7c286030dde0_characterizeImage_1101466_44.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/a8825bcc-a28b-40ca-9f20-7c286030dde0_characterizeImage_1101466_44.stderr parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf79f52df4cda9e1bf26a8a8c98d2dc8 with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash bd1778f9a47f3c9c12b799434fd491a3 parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b76de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/07fdbf39-07c2-4865-bb1d-05ea985f165e_calibrate_1101466_28.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/07fdbf39-07c2-4865-bb1d-05ea985f165e_calibrate_1101466_28.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/138bde30-b0bc-4618-9795-9910de1c2513_characterizeImage_1101466_28.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/138bde30-b0bc-4618-9795-9910de1c2513_characterizeImage_1101466_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:43 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 039b784095960f6d5de2f90e5925aa52 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 52348789dd8f170c04b4a11308e25eb5 parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a05440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/c2f7e6bd-5b74-4a94-9b4e-d860ecd7ddb8_calibrate_1101469_31.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/c2f7e6bd-5b74-4a94-9b4e-d860ecd7ddb8_calibrate_1101469_31.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/be62030f-d331-491f-aab4-a343704a5125_characterizeImage_1101469_31.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/be62030f-d331-491f-aab4-a343704a5125_characterizeImage_1101469_31.stderr parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c13058c2d62e86247a949d9cf43c313 with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 84994da9792a187af7ad0df1d87718ce parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/cf1ca470-8238-4e5f-8318-e7b203cd7af4_calibrate_1101466_54.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/cf1ca470-8238-4e5f-8318-e7b203cd7af4_calibrate_1101466_54.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e953d57e-c07d-4457-90fd-8134696a2907_characterizeImage_1101466_54.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e953d57e-c07d-4457-90fd-8134696a2907_characterizeImage_1101466_54.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5ec35425a7c6c4fc8659628016f2e4d with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 8eb6543b042d222fcb31ff9df5a479a6 parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b38e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/f3ef0878-f074-4db2-9bfa-5feccddda07f_calibrate_1101475_38.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/f3ef0878-f074-4db2-9bfa-5feccddda07f_calibrate_1101475_38.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/9022c339-1bbc-435a-98b4-a0c60c350907_characterizeImage_1101475_38.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/9022c339-1bbc-435a-98b4-a0c60c350907_characterizeImage_1101475_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:44 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3b9be754f66f005c748407808f30eb8 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash f6116cfb4fd4261755e345a690fe55c0 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b38ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/c3c3ae2b-7519-4f8b-8044-5e477fa56f27_calibrate_1101472_42.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/c3c3ae2b-7519-4f8b-8044-5e477fa56f27_calibrate_1101472_42.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/cfa2a5be-826d-4085-a33b-f25bb0012d32_characterizeImage_1101472_42.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/cfa2a5be-826d-4085-a33b-f25bb0012d32_characterizeImage_1101472_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:46 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93500ea1284c69cb4b238c4e3cce6257 with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 830828e1d7b9570545d07f455569191a parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a05580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/24540e2f-9e59-4e46-9891-9dcc504c173e_calibrate_1101475_32.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/24540e2f-9e59-4e46-9891-9dcc504c173e_calibrate_1101475_32.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8ffc3177-1cca-414d-9f83-72d28f4791ef_characterizeImage_1101475_32.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8ffc3177-1cca-414d-9f83-72d28f4791ef_characterizeImage_1101475_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:47 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71df7c13a8a53655de63e98526402d56 with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 376c0a251c09bcb26c81c6f291658253 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/11f5f2ec-9c78-4309-bb79-c9d5813291a0_calibrate_1101466_5.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/11f5f2ec-9c78-4309-bb79-c9d5813291a0_calibrate_1101466_5.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/327665bb-94fb-4680-82d2-9f5c7f060273_characterizeImage_1101466_5.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/327665bb-94fb-4680-82d2-9f5c7f060273_characterizeImage_1101466_5.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f871640ad9a9163a7853d05c98619f5e with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 463ba582be891a99eedc64a4ed3b2677 parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b77240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/270d0153-1458-4aa1-9432-4386b076adef_calibrate_1101466_56.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/270d0153-1458-4aa1-9432-4386b076adef_calibrate_1101466_56.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/5355e111-3c40-4608-9108-42210a716f0c_characterizeImage_1101466_56.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/5355e111-3c40-4608-9108-42210a716f0c_characterizeImage_1101466_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:49 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1006f3bae99f8a18f36aea71eec91ac with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash e4bfb0ec4c74a329352f2e200140b198 parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b04c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/b5ba0dea-c940-48b4-a7a7-95c68638c29a_calibrate_1101472_28.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/b5ba0dea-c940-48b4-a7a7-95c68638c29a_calibrate_1101472_28.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d9a5ca31-6c67-481c-92d0-9db7743b5e77_characterizeImage_1101472_28.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d9a5ca31-6c67-481c-92d0-9db7743b5e77_characterizeImage_1101472_28.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a9ded743542a90f85946e3cb2f952c0 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 8fccd2057f4dd02111c8824f71cffb29 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/126ecf71-70dd-4552-84b9-bc76346b98dc_calibrate_1101472_14.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/126ecf71-70dd-4552-84b9-bc76346b98dc_calibrate_1101472_14.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/58eebfb7-c355-4ee4-9719-5fbceaf26821_characterizeImage_1101472_14.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/58eebfb7-c355-4ee4-9719-5fbceaf26821_characterizeImage_1101472_14.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 400facc4c6ac790bd1f1e33326a7778d with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 011e3e3515930d392c0fced4f5cae13b parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c8ebcf9d-0acb-4e7b-a72c-23af62adf83b_calibrate_1101475_1.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c8ebcf9d-0acb-4e7b-a72c-23af62adf83b_calibrate_1101475_1.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/16646fea-60a9-410c-9c59-a7a42e34a1ed_characterizeImage_1101475_1.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/16646fea-60a9-410c-9c59-a7a42e34a1ed_characterizeImage_1101475_1.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 038c8b3b337271993289a025a2ee923f with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 8f3d16e04515ce2d5258ac26aa5944ed parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/90feaf23-93c7-49bb-8e4b-a408b1b96b15_calibrate_1101469_48.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/90feaf23-93c7-49bb-8e4b-a408b1b96b15_calibrate_1101469_48.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/47fb1d5d-5d06-47eb-9772-7d3c09839765_characterizeImage_1101469_48.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/47fb1d5d-5d06-47eb-9772-7d3c09839765_characterizeImage_1101469_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:50 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:52 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:54 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 802e1dee81f7c6bb0f3a8cf18a2c7876 with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 7008ba897cf1ef7a164c151b7b7930c2 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a063e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/217e953d-b0d9-4d98-9926-f5d53cfd172a_calibrate_1101466_40.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/217e953d-b0d9-4d98-9926-f5d53cfd172a_calibrate_1101466_40.stderr parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7588e5ec-5039-4385-b4a1-4dfe6bc09282_characterizeImage_1101466_40.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7588e5ec-5039-4385-b4a1-4dfe6bc09282_characterizeImage_1101466_40.stderr parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99d88ca520c345227fa454c5498ce47c with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 2da41e965c20559a3ab9fc0f1d2462da parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/686bd3f0-b736-49c1-81aa-02c172060228_calibrate_1101472_27.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/686bd3f0-b736-49c1-81aa-02c172060228_calibrate_1101472_27.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/b61b7a75-af1a-4de9-aacc-ddf934a7364e_characterizeImage_1101472_27.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/b61b7a75-af1a-4de9-aacc-ddf934a7364e_characterizeImage_1101472_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:55 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de12bf27d045bcf716024696369f6cfa with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 5cb431c8a20183c798cd64142f92fd9c parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609beca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/0e52825c-5be6-43f3-8909-81f30f3f7336_calibrate_1101472_20.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/0e52825c-5be6-43f3-8909-81f30f3f7336_calibrate_1101472_20.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/912a9589-7166-4b84-a638-2ebc8a3cc84f_characterizeImage_1101472_20.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/912a9589-7166-4b84-a638-2ebc8a3cc84f_characterizeImage_1101472_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:57 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ed2ace7f4307839336ab16b09949170 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash 8609dbeb7ec25c6c888fb8c5e5432243 parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/bc2d84e2-1ea5-46e4-a74d-3a82af88b246_calibrate_1101469_44.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/bc2d84e2-1ea5-46e4-a74d-3a82af88b246_calibrate_1101469_44.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/f6cf8b98-8b9b-4284-bc24-52ceb3a3ed06_characterizeImage_1101469_44.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/f6cf8b98-8b9b-4284-bc24-52ceb3a3ed06_characterizeImage_1101469_44.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ded58828fc819f0ae4e01c6f896fc3b with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 025eb22ee9de74bf472df6323ffd4589 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/608b444e-2983-478c-b0fe-e037192f4982_calibrate_1101466_41.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/608b444e-2983-478c-b0fe-e037192f4982_calibrate_1101466_41.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/612de6a8-b7cf-4cd9-a5b9-c2b77805e11a_characterizeImage_1101466_41.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/612de6a8-b7cf-4cd9-a5b9-c2b77805e11a_characterizeImage_1101466_41.stderr parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ad7b58246fa3dcdf87bdb4612cbfeda with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash deb56efd678818302b629acd1651ca10 parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d02fac2f-250c-413b-9a50-f48ea958eb5a_calibrate_1101472_57.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d02fac2f-250c-413b-9a50-f48ea958eb5a_calibrate_1101472_57.stderr parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/56b73c35-0fbc-4339-bb4c-09e87b4c58b5_characterizeImage_1101472_57.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/56b73c35-0fbc-4339-bb4c-09e87b4c58b5_characterizeImage_1101472_57.stderr parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98b7cdc8dab40c78f530106c3b57b4e9 with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 77eb61e261a7176d9d7f130239a131b0 parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/bc9c491e-2759-44be-a996-c329df0a7257_calibrate_1101466_55.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/bc9c491e-2759-44be-a996-c329df0a7257_calibrate_1101466_55.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/87bfe706-b586-439a-9923-b8e8edcc2b8b_characterizeImage_1101466_55.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/87bfe706-b586-439a-9923-b8e8edcc2b8b_characterizeImage_1101466_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:05:59 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58e021e54ef6b98c0349b3357bc2840b with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 0679b19aa23620b5979672a11c81442e parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/9ac1aa47-cf5d-43dc-a46e-398673458c58_calibrate_1101475_8.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/9ac1aa47-cf5d-43dc-a46e-398673458c58_calibrate_1101475_8.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c891c68f-13e2-4951-9fac-ba72794009e5_characterizeImage_1101475_8.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c891c68f-13e2-4951-9fac-ba72794009e5_characterizeImage_1101475_8.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46be3a42139acecb7f8b5abb5d4d3cf5 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 6895b2efafea44bdf7ec0ce20d1f4437 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c382c125-ef19-4db1-9de7-c277b49e5c8e_calibrate_1101475_26.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c382c125-ef19-4db1-9de7-c277b49e5c8e_calibrate_1101475_26.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/d577b439-7712-4652-9ff0-4d8e230f6ccb_characterizeImage_1101475_26.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/d577b439-7712-4652-9ff0-4d8e230f6ccb_characterizeImage_1101475_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:01 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfbe47c827048406d1ee430cdc2226b9 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 632bc667be3817137e91a4742d063955 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/f09d1ab1-3d2b-4631-b6cc-b9d355c1bbba_calibrate_1101469_3.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/f09d1ab1-3d2b-4631-b6cc-b9d355c1bbba_calibrate_1101469_3.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/b5763be8-14fc-4be2-a503-d50f71d4029e_characterizeImage_1101469_3.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/b5763be8-14fc-4be2-a503-d50f71d4029e_characterizeImage_1101469_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:04 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ceed0d37a716f8c7531a3b9549a90bf with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 4e714a9e3d3f7a75f86473d7e9460a33 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b74860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2d46ea12-6516-4ed6-be76-7904ffeebffa_calibrate_1101466_24.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2d46ea12-6516-4ed6-be76-7904ffeebffa_calibrate_1101466_24.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/f9794297-4986-4d71-8ae9-66f09f45752c_characterizeImage_1101466_24.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/f9794297-4986-4d71-8ae9-66f09f45752c_characterizeImage_1101466_24.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 294cf19d3b8062f7cc9f0775772afba6 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 9cd15cce77161f000931a0d9dced2d2d parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b76480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/652af4a9-8b60-4386-a72f-606540fe7df0_calibrate_1101472_9.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/652af4a9-8b60-4386-a72f-606540fe7df0_calibrate_1101472_9.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/cb168b21-0cd0-4727-8800-9bc1b9aa8bc7_characterizeImage_1101472_9.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/cb168b21-0cd0-4727-8800-9bc1b9aa8bc7_characterizeImage_1101472_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:07 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a206fcaf1af9b655878ffe5c665c4fa8 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 1b614bc7c3941666432ed799af59b886 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b071a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/c81dc57e-5ecc-418a-91cc-854eaf29d85f_calibrate_1101469_25.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/c81dc57e-5ecc-418a-91cc-854eaf29d85f_calibrate_1101469_25.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/0a1c9f0f-028a-49c8-86cd-c597fd38305a_characterizeImage_1101469_25.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/0a1c9f0f-028a-49c8-86cd-c597fd38305a_characterizeImage_1101469_25.stderr parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea3feb77119b448074418b6b84b0f784 with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash 5f8a1cc49116765051d31c2bd09b7064 parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/aa558952-0c75-43ab-9d40-3318703728a0_calibrate_1101466_13.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/aa558952-0c75-43ab-9d40-3318703728a0_calibrate_1101466_13.stderr parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/17059bde-0e36-49e7-ae34-dd4ec916e45b_characterizeImage_1101466_13.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/17059bde-0e36-49e7-ae34-dd4ec916e45b_characterizeImage_1101466_13.stderr parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac36ab023585f76f3bda4f0f9fea6d24 with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash b5ad639b71e525cc73fadfbd3660c8df parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bdbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/43aff674-0e1e-442b-ac27-4ff219d26b3e_calibrate_1101475_19.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/43aff674-0e1e-442b-ac27-4ff219d26b3e_calibrate_1101475_19.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a32dfb08-cedc-4588-aac6-ed45b0c54557_characterizeImage_1101475_19.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a32dfb08-cedc-4588-aac6-ed45b0c54557_characterizeImage_1101475_19.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef427fb7819345d8ea635ac0c3a34429 with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash 67073374039372c50d531337a03f5de3 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/7c5ebb93-35f5-4e66-a54b-e07f759a0a2d_calibrate_1101466_11.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/7c5ebb93-35f5-4e66-a54b-e07f759a0a2d_calibrate_1101466_11.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/9ae22180-cc92-41bb-9b99-fbb49ae5304a_characterizeImage_1101466_11.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/9ae22180-cc92-41bb-9b99-fbb49ae5304a_characterizeImage_1101466_11.stderr parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de5e78a17cfbc8266ceb85c72901aaf1 with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash ec57f01c5548b6b5f4b9538744010da4 parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b76ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/01bdd519-625d-4be3-9126-63a4b0acfd80_calibrate_1101466_10.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/01bdd519-625d-4be3-9126-63a4b0acfd80_calibrate_1101466_10.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/17ba1955-00cb-447e-a5ef-a558f4ec61f1_characterizeImage_1101466_10.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/17ba1955-00cb-447e-a5ef-a558f4ec61f1_characterizeImage_1101466_10.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ff3299a7b0b953900c7b81e277c210c with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 8c480576e44668ddca5bbe9e4cc5cb8d parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/9ee46f26-ade8-42ac-b8e9-4ade7ad4f473_calibrate_1101472_23.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/9ee46f26-ade8-42ac-b8e9-4ade7ad4f473_calibrate_1101472_23.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/a33085e2-63c6-4d01-bc41-c39f6b65c7e2_characterizeImage_1101472_23.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/a33085e2-63c6-4d01-bc41-c39f6b65c7e2_characterizeImage_1101472_23.stderr parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4164cccb4001a40443feb32e509c691c with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash 1f57f29dc1c14f56e85a1e230813754b parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/e2a30825-1f2a-4ac0-a516-e269344f0824_calibrate_1101466_38.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/e2a30825-1f2a-4ac0-a516-e269344f0824_calibrate_1101466_38.stderr parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/8d1193e2-641b-45a3-a6c1-da4fcba36527_characterizeImage_1101466_38.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/8d1193e2-641b-45a3-a6c1-da4fcba36527_characterizeImage_1101466_38.stderr parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a80a271a1b126833fd044793425c4f1 with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 6dd7f79ccd81d9d4ea53f530aca3b46f parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/17cdd4a9-80c8-4ba7-830a-245b040380ea_calibrate_1101472_8.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/17cdd4a9-80c8-4ba7-830a-245b040380ea_calibrate_1101472_8.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d296b2fa-eb6c-43aa-a78c-d44c0caaaf38_characterizeImage_1101472_8.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d296b2fa-eb6c-43aa-a78c-d44c0caaaf38_characterizeImage_1101472_8.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfe637077da18149271e992fbd4454a6 with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash f5b9b241041ed1f31878ac6a87b0a2b6 parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b77600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/7799bbae-8aa1-4061-bc65-d6bd09721a50_calibrate_1101472_22.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/7799bbae-8aa1-4061-bc65-d6bd09721a50_calibrate_1101472_22.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4911fa5d-1094-4f14-835b-a2cf632e94fd_characterizeImage_1101472_22.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4911fa5d-1094-4f14-835b-a2cf632e94fd_characterizeImage_1101472_22.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bf8f9e5529477c51e4c4cbd46a23bdc with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 90b3e10f7ec57d0783ce16e1b6a4bc67 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/24ec0977-5300-4de1-9f63-132425c433e2_calibrate_1101475_17.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/24ec0977-5300-4de1-9f63-132425c433e2_calibrate_1101475_17.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/db963836-3798-4422-9161-29e8be316af9_characterizeImage_1101475_17.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/db963836-3798-4422-9161-29e8be316af9_characterizeImage_1101475_17.stderr parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02091856f499129ec29d1cacde274965 with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash b21eeca255a67693a8f924a7abd4db17 parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/da47c8e3-040f-4620-8d46-6dfc1f3d0fc0_calibrate_1101469_39.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/da47c8e3-040f-4620-8d46-6dfc1f3d0fc0_calibrate_1101469_39.stderr parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/d1411d97-da67-4d6e-802d-2d6f766424e6_characterizeImage_1101469_39.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/d1411d97-da67-4d6e-802d-2d6f766424e6_characterizeImage_1101469_39.stderr parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e78e931239f603b32d713fb75baef07f with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash b267e9b93227ec182a7142e9b41b1a46 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/bc1b4c09-27ad-4413-8a8b-61b7d66aff56_calibrate_1101469_10.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/bc1b4c09-27ad-4413-8a8b-61b7d66aff56_calibrate_1101469_10.stderr parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c6d41c88-73e8-445f-b15c-6276ecf551c8_characterizeImage_1101469_10.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c6d41c88-73e8-445f-b15c-6276ecf551c8_characterizeImage_1101469_10.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 145 try 1 failed parsl.dataflow.dflow ERROR: Task 145 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 33baa97308adde9ae27015e5231efd7d with result from task 145 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 146 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 146 try 0 failed parsl.dataflow.dflow INFO: Task 146 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4cf91f95-1462-4865-a991-c47564eb06f7_calibrate_1101466_2.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4cf91f95-1462-4865-a991-c47564eb06f7_calibrate_1101466_2.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7c79a3c6-e56d-4b40-bd06-8eb914dec934_characterizeImage_1101466_2.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7c79a3c6-e56d-4b40-bd06-8eb914dec934_characterizeImage_1101466_2.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 446b390adfa937487a383faa5bfab2b0 with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash 7deda1e358722f86505e034674619eeb parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/3930ab43-b324-42b3-83b6-a3dc580d6033_calibrate_1101475_62.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/3930ab43-b324-42b3-83b6-a3dc580d6033_calibrate_1101475_62.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/01be6faf-3406-4ae6-a019-2de1ae57f5b5_characterizeImage_1101475_62.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/01be6faf-3406-4ae6-a019-2de1ae57f5b5_characterizeImage_1101475_62.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8b2b274a948b8ea742cea2ed3bca543 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash cdb0918be66b8f4a3b9caa184d04a407 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b06e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/9874250b-d333-4f8b-b3b6-80a190ce549c_calibrate_1101472_17.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/9874250b-d333-4f8b-b3b6-80a190ce549c_calibrate_1101472_17.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4014b455-e6ca-4426-b428-73df35fe99b8_characterizeImage_1101472_17.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4014b455-e6ca-4426-b428-73df35fe99b8_characterizeImage_1101472_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:16 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b0a04fa4aee5c0d15c5ef95e3dfac8a with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash 102a1efa3b858047319fbe59711e4905 parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8a473e4e-206a-4140-a75c-17b5eb51c83c_calibrate_1101469_49.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8a473e4e-206a-4140-a75c-17b5eb51c83c_calibrate_1101469_49.stderr parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/094998b6-3adf-4aa8-abf0-c0cbbadb6985_characterizeImage_1101469_49.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/094998b6-3adf-4aa8-abf0-c0cbbadb6985_characterizeImage_1101469_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:18 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69c6ee7dbb7520b769d65c0365a65c3d with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 4a4202f85ee8f60fb0300c3b2211e971 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146162075e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/332ccfd3-2323-4a4f-a7ad-5db8c2d879b3_calibrate_1101472_36.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/332ccfd3-2323-4a4f-a7ad-5db8c2d879b3_calibrate_1101472_36.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/a5541e60-e8c8-46ed-a7cd-9e7539ba89a0_characterizeImage_1101472_36.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/a5541e60-e8c8-46ed-a7cd-9e7539ba89a0_characterizeImage_1101472_36.stderr parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01e4d546b48958a0d5658d118ff2c833 with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash 25f809a8f60a37e00d3dd781b1ffa7ef parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/9b9026a4-c866-4d0e-beb4-0cc7abc7f1da_calibrate_1101469_35.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/9b9026a4-c866-4d0e-beb4-0cc7abc7f1da_calibrate_1101469_35.stderr parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/abc03a38-f803-481d-b73a-c273cbd6bbda_characterizeImage_1101469_35.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/abc03a38-f803-481d-b73a-c273cbd6bbda_characterizeImage_1101469_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:19 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:21 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54e9a440d4a0819de97b3c80a98ba6df with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 9437b9ef7aa9a54a86a09b6023722ed3 parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2d408b9f-6fd6-4a76-b243-263c47a43ed0_calibrate_1101466_32.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2d408b9f-6fd6-4a76-b243-263c47a43ed0_calibrate_1101466_32.stderr parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/348bc317-d6f8-4f8b-a807-8a5e0cf4fb61_characterizeImage_1101466_32.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/348bc317-d6f8-4f8b-a807-8a5e0cf4fb61_characterizeImage_1101466_32.stderr parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a40e02e7c1ed2b35b8228540b66f2bc with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash a742506d55143c6b248982d381735c23 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a05760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/67a75334-339a-4466-916a-893485b53f23_calibrate_1101472_30.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/67a75334-339a-4466-916a-893485b53f23_calibrate_1101472_30.stderr parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/c706ce05-da4c-4940-b1e3-1e90357a1a22_characterizeImage_1101472_30.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/c706ce05-da4c-4940-b1e3-1e90357a1a22_characterizeImage_1101472_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76a3446e151d475c84412a75f3b75553 with result from task 328 parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:22 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 0cf68ce7dd4f6a2f90245a5022f55951 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b06520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/41421b1e-fa98-4226-82a3-adba83b31982_calibrate_1101475_6.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/41421b1e-fa98-4226-82a3-adba83b31982_calibrate_1101475_6.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/cf974039-f615-48e6-85e5-4d235a146327_characterizeImage_1101475_6.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/cf974039-f615-48e6-85e5-4d235a146327_characterizeImage_1101475_6.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 784236897a757a4b7d2cb4214ddebc17 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 2736030b796e8025e140424058c41443 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b058a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/383f8bbe-2aa2-47db-bf16-d0ff907555ae_calibrate_1101475_36.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/383f8bbe-2aa2-47db-bf16-d0ff907555ae_calibrate_1101475_36.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/9f913d4a-5cbc-4b01-b9a6-8277764eac4f_characterizeImage_1101475_36.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/9f913d4a-5cbc-4b01-b9a6-8277764eac4f_characterizeImage_1101475_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:23 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14e397b7660a6b837fc772f38b143278 with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 757ae3907ceea09ea000cdea0c145745 parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a067a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d7f0a449-1fae-4599-af24-c5c19516340e_calibrate_1101472_50.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d7f0a449-1fae-4599-af24-c5c19516340e_calibrate_1101472_50.stderr parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/02d90e95-bf7a-4263-8e77-441c3de63973_characterizeImage_1101472_50.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/02d90e95-bf7a-4263-8e77-441c3de63973_characterizeImage_1101472_50.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 25 try 1 failed parsl.dataflow.dflow ERROR: Task 25 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry aeb0ceee47fe18cbd564a4fc3350daf1 with result from task 25 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 26 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 26 try 0 failed parsl.dataflow.dflow INFO: Task 26 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/d1a4579d-ba26-47b1-b204-70328154f8c3_calibrate_1101466_61.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/d1a4579d-ba26-47b1-b204-70328154f8c3_calibrate_1101466_61.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/df2478d3-b391-4d93-a2cd-a87ad591e22a_characterizeImage_1101466_61.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/df2478d3-b391-4d93-a2cd-a87ad591e22a_characterizeImage_1101466_61.stderr parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3dbd069d7b4cb5c3e39662981882c15 with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash c91c9f5da174742915b2c9f0a32bbda0 parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609be5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/c3e613c8-f4e4-49d9-a558-b730d087f1be_calibrate_1101472_55.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/c3e613c8-f4e4-49d9-a558-b730d087f1be_calibrate_1101472_55.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/ed2db7f7-40b3-4c67-86cb-2194a072bab5_characterizeImage_1101472_55.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/ed2db7f7-40b3-4c67-86cb-2194a072bab5_characterizeImage_1101472_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:25 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11f9593219ffc9adf5a701ac41913a5f with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash dde90ccce3e475383a4e9c01fe819745 parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/9d6e4339-020b-40f8-a1fc-80843588d9a3_calibrate_1101469_54.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/9d6e4339-020b-40f8-a1fc-80843588d9a3_calibrate_1101469_54.stderr parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/538a3462-ee07-465c-8ee3-e2ff7b8ca249_characterizeImage_1101469_54.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/538a3462-ee07-465c-8ee3-e2ff7b8ca249_characterizeImage_1101469_54.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b48e92cc58d62b9d9330aabe05342ea with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash 4f48b0825ae177a40131f62512754547 parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/863d401d-16a4-46a8-afbe-78b20704607a_calibrate_1101466_8.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/863d401d-16a4-46a8-afbe-78b20704607a_calibrate_1101466_8.stderr parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/762e74c3-40d8-46ca-ac75-aa33436361f4_characterizeImage_1101466_8.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/762e74c3-40d8-46ca-ac75-aa33436361f4_characterizeImage_1101466_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:27 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.memoization DEBUG: Storing app cache entry e3627242cf0419c89a694da7345b8873 with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash b0cea21da5f90af48aab73ac120ac264 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a04e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/0e12f3b2-4aa3-43e6-827e-4d2cb05d86c9_calibrate_1101469_42.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/0e12f3b2-4aa3-43e6-827e-4d2cb05d86c9_calibrate_1101469_42.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/76b7c889-006a-4a42-8110-115ff2bc4d67_characterizeImage_1101469_42.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/76b7c889-006a-4a42-8110-115ff2bc4d67_characterizeImage_1101469_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:28 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:30 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2ae1dee939f2c1a478247ad135615e6 with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 1db55163ff6f787b016da39ae32d3869 parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/6f762e54-d73e-4dae-ae66-edd2a5c3ab44_calibrate_1101466_7.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/6f762e54-d73e-4dae-ae66-edd2a5c3ab44_calibrate_1101466_7.stderr parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/75859e87-bd40-4928-8508-bcc3d5ee2213_characterizeImage_1101466_7.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/75859e87-bd40-4928-8508-bcc3d5ee2213_characterizeImage_1101466_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:32 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 295 try 1 failed parsl.dataflow.dflow ERROR: Task 295 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ff0c04f45cf2d2704887381b773a02b2 with result from task 295 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 296 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 296 try 0 failed parsl.dataflow.dflow INFO: Task 296 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2cc440e4-d8f0-4248-80b4-3d2f48c4fcac_calibrate_1101469_2.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2cc440e4-d8f0-4248-80b4-3d2f48c4fcac_calibrate_1101469_2.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2500a4d4-f34a-47e2-a1f8-8543d55c832d_characterizeImage_1101469_2.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2500a4d4-f34a-47e2-a1f8-8543d55c832d_characterizeImage_1101469_2.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1346181c18d7c64285cdb7a21c505be9 with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 4692d001bb1f99e86d9438bbd439ff0a parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b05ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/228eee85-f709-4e47-a918-a68898160c9c_calibrate_1101472_16.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/228eee85-f709-4e47-a918-a68898160c9c_calibrate_1101472_16.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/bf0c1ec6-e75e-4737-b59f-a824a49589dc_characterizeImage_1101472_16.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/bf0c1ec6-e75e-4737-b59f-a824a49589dc_characterizeImage_1101472_16.stderr parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1830bcd77900e4722c0b8cec911bc3cc with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash ab2d4dd770a7a572c3386c6dfe6b2641 parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/f0cdf252-da50-4e3a-8639-7918d4719af3_calibrate_1101466_39.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/f0cdf252-da50-4e3a-8639-7918d4719af3_calibrate_1101466_39.stderr parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/63f662ea-585e-4dee-8824-f666c97740b7_characterizeImage_1101466_39.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/63f662ea-585e-4dee-8824-f666c97740b7_characterizeImage_1101466_39.stderr parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0474a85fc4e8c91821adb378376ffba with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash a5119aa3666105019c8aac6c9c3f1d09 parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608efb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/588a91f0-75db-4ffc-b7b9-4a962db227cf_calibrate_1101466_15.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/588a91f0-75db-4ffc-b7b9-4a962db227cf_calibrate_1101466_15.stderr parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/8d519ac7-61ca-4bf9-adcc-30195ea3a889_characterizeImage_1101466_15.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/8d519ac7-61ca-4bf9-adcc-30195ea3a889_characterizeImage_1101466_15.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be0b0f9be47063db9fa119c7431e7acd with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 673b4477a784891864f57ec3e8da4807 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160adc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/14437f3a-b890-4597-8758-c163ee4a32f5_calibrate_1101475_22.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/14437f3a-b890-4597-8758-c163ee4a32f5_calibrate_1101475_22.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/ffcb7d03-1e1f-4398-9e27-4398ff459ae9_characterizeImage_1101475_22.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/ffcb7d03-1e1f-4398-9e27-4398ff459ae9_characterizeImage_1101475_22.stderr parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab8d28e9d9958c0c694037d8f8eaa032 with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash 8469e505aa1a41be15f9894a8600531c parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4828693e-e189-4d8e-91d4-84cef0997bcc_calibrate_1101466_59.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4828693e-e189-4d8e-91d4-84cef0997bcc_calibrate_1101466_59.stderr parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/dad1cbc9-8497-4c32-becf-6d3cd67c6c4e_characterizeImage_1101466_59.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/dad1cbc9-8497-4c32-becf-6d3cd67c6c4e_characterizeImage_1101466_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:34 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 878d404e090385d0c431bf277aae9d5d with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 24bf51bdc0a23ce4487414bd6612036c parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/94f93ecf-3350-4dde-89c9-f641b48d17f8_calibrate_1101475_12.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/94f93ecf-3350-4dde-89c9-f641b48d17f8_calibrate_1101475_12.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c3eebaaa-b67e-4939-a656-6762a3b95905_characterizeImage_1101475_12.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c3eebaaa-b67e-4939-a656-6762a3b95905_characterizeImage_1101475_12.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9411714248360c11fce0129036ad6611 with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash 964165677bb768db7e827267562535db parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d47b7a3b-8d8c-4b86-97c0-147014599a64_calibrate_1101472_7.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d47b7a3b-8d8c-4b86-97c0-147014599a64_calibrate_1101472_7.stderr parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/7f2df31e-9dec-48d2-9261-d2307ddfa367_characterizeImage_1101472_7.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/7f2df31e-9dec-48d2-9261-d2307ddfa367_characterizeImage_1101472_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:36 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7130e4d1681d3f911444f3a16eecea3 with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash 08aec3ff51b10a0b46378c9818a2326c parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ec540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/b20586e2-aa0f-4d80-b797-9c8f95ee3ec2_calibrate_1101469_53.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/b20586e2-aa0f-4d80-b797-9c8f95ee3ec2_calibrate_1101469_53.stderr parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/72d36d32-5d23-4ec3-9237-abbf3c14d089_characterizeImage_1101469_53.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/72d36d32-5d23-4ec3-9237-abbf3c14d089_characterizeImage_1101469_53.stderr parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2f8e336b7ca160285845ae81dca7a80 with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash dd2b5afb23048bd02209ba42892f36bb parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/73216ea5-b441-4e43-866d-204522a7587f_calibrate_1101475_15.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/73216ea5-b441-4e43-866d-204522a7587f_calibrate_1101475_15.stderr parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/78dd6562-5636-46b5-bc11-1a5ec69bdeb2_characterizeImage_1101475_15.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/78dd6562-5636-46b5-bc11-1a5ec69bdeb2_characterizeImage_1101475_15.stderr parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eba4c069785fce6140e6206dcc3f4c41 with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash 027f8b4bdb4a8c059c0fb3431ffc401e parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a05c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7f6238ba-a64a-4904-bcbb-646d97f21dcc_calibrate_1101469_55.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7f6238ba-a64a-4904-bcbb-646d97f21dcc_calibrate_1101469_55.stderr parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/255a48e2-0aa9-4326-b6ed-7b12518b2f4a_characterizeImage_1101469_55.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/255a48e2-0aa9-4326-b6ed-7b12518b2f4a_characterizeImage_1101469_55.stderr parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e38e7219035cbc06e84fb94e1396b58 with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash 9fb47b40ce8d0d8954e23938047d03e2 parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/1c9758cb-ea60-4b8b-b4e8-cea5701c78e8_calibrate_1101472_5.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/1c9758cb-ea60-4b8b-b4e8-cea5701c78e8_calibrate_1101472_5.stderr parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/fa9c43b7-1dbd-4024-8f4c-9f0b3f4467ab_characterizeImage_1101472_5.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/fa9c43b7-1dbd-4024-8f4c-9f0b3f4467ab_characterizeImage_1101472_5.stderr parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0922f679b503b503ee3c02436e2b1c57 with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash aa540612811a4f2b354af992122ef1fd parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/4bbfc788-17a7-404b-8f46-adda138f1fbd_calibrate_1101469_4.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/4bbfc788-17a7-404b-8f46-adda138f1fbd_calibrate_1101469_4.stderr parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/5f7f32c2-9567-4f1e-8623-8676cf680541_characterizeImage_1101469_4.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/5f7f32c2-9567-4f1e-8623-8676cf680541_characterizeImage_1101469_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:38 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d4525dfe935e3c3dec2709f347577d0 with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 2cd625dda284d88f0c88b81876ea3174 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/8ab3f1d8-70f5-49b1-aa84-9ce17dc4b8b4_calibrate_1101466_23.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/8ab3f1d8-70f5-49b1-aa84-9ce17dc4b8b4_calibrate_1101466_23.stderr parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7c18b06e-733b-4915-ac61-ad00b68f3015_characterizeImage_1101466_23.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/7c18b06e-733b-4915-ac61-ad00b68f3015_characterizeImage_1101466_23.stderr parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6e2527911b37feb2a0d02de86d89f98 with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash e294515a632ae4498fbf54fb691c64fb parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2f93d67a-369a-4dee-b32b-9cc8986514c1_calibrate_1101469_1.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2f93d67a-369a-4dee-b32b-9cc8986514c1_calibrate_1101469_1.stderr parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/21cddd1d-8494-4d8d-965f-12e413387de7_characterizeImage_1101469_1.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/21cddd1d-8494-4d8d-965f-12e413387de7_characterizeImage_1101469_1.stderr parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b7e8d58b4fc7ae7e3cbb7f8a9f4147b with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 98a4ccac1cceb7aaf862415e28dfe7cc parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ecb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/5ba8537f-7b5f-46df-85aa-462e690820d9_calibrate_1101472_47.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/5ba8537f-7b5f-46df-85aa-462e690820d9_calibrate_1101472_47.stderr parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4c0ddab0-a20f-4dd5-a3be-39f8d5e69664_characterizeImage_1101472_47.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4c0ddab0-a20f-4dd5-a3be-39f8d5e69664_characterizeImage_1101472_47.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fcedd3ddfd306e163b20d5d24245134 with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash 0c35c660bb135ad292f3c27388400a2d parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2d35e606-6052-4db7-9aed-ffb04a8576ba_calibrate_1101469_24.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2d35e606-6052-4db7-9aed-ffb04a8576ba_calibrate_1101469_24.stderr parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/524ecf8d-1ef2-496e-86a1-9c1ac470db29_characterizeImage_1101469_24.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/524ecf8d-1ef2-496e-86a1-9c1ac470db29_characterizeImage_1101469_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:39 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:41 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a63136167703d5c7ca4e267928e79c86 with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash dccd67df71106c60e7238de14f6883a6 parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/ac7f3fb2-10f9-45a3-9dc8-76dd51a6527c_calibrate_1101475_20.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/ac7f3fb2-10f9-45a3-9dc8-76dd51a6527c_calibrate_1101475_20.stderr parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/cc01c6fe-c1aa-4b92-b6cc-855a9a262deb_characterizeImage_1101475_20.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/cc01c6fe-c1aa-4b92-b6cc-855a9a262deb_characterizeImage_1101475_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:43 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8f3832366b9d1e3f127efc62adabe99 with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash 0e66df33a9f27a2d27e1f8360961eeab parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/0e2eaafc-f7ee-42da-ab1c-5a829b9eda05_calibrate_1101475_43.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/0e2eaafc-f7ee-42da-ab1c-5a829b9eda05_calibrate_1101475_43.stderr parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/96b4e15f-b2fd-4520-8edc-4183a26342b4_characterizeImage_1101475_43.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/96b4e15f-b2fd-4520-8edc-4183a26342b4_characterizeImage_1101475_43.stderr parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f35ae243f8d3517191e4e1975347be52 with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash 232ca84d35f36dd327ae9d29c4854392 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ecea88b8-4629-47a1-a6bf-3b38e9f9b245_calibrate_1101472_26.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ecea88b8-4629-47a1-a6bf-3b38e9f9b245_calibrate_1101472_26.stderr parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/ca1ee52b-97c8-4e8b-bb0c-7e01e5143e51_characterizeImage_1101472_26.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/ca1ee52b-97c8-4e8b-bb0c-7e01e5143e51_characterizeImage_1101472_26.stderr parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5596728118ec0dba43ddf84a1a6d4081 with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash 416178ea005970cf8dc87cca41050a92 parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a06a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/90df92bc-a3ea-4c67-9d60-abd8c26e4b73_calibrate_1101472_35.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/90df92bc-a3ea-4c67-9d60-abd8c26e4b73_calibrate_1101472_35.stderr parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/b2345076-cd56-4ef7-8f5c-898e202a9672_characterizeImage_1101472_35.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/b2345076-cd56-4ef7-8f5c-898e202a9672_characterizeImage_1101472_35.stderr parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1dcd0f0f774312b59a1560a395901b2 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash 4ad44118e1d48e0ed7c7667a8b78c586 parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4500bbcf-6ca9-48c1-986b-3afda3084d99_calibrate_1101466_9.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4500bbcf-6ca9-48c1-986b-3afda3084d99_calibrate_1101466_9.stderr parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c65fa402-599a-4790-b575-f77a3e367532_characterizeImage_1101466_9.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c65fa402-599a-4790-b575-f77a3e367532_characterizeImage_1101466_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:44 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 538 try 1 failed parsl.dataflow.dflow ERROR: Task 538 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 77b012388d3e1bae2ab72b37985a85a1 with result from task 538 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 539 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 539 try 0 failed parsl.dataflow.dflow INFO: Task 539 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/28f13319-aaee-499e-9faf-68b71dc1b673_calibrate_1101475_2.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/28f13319-aaee-499e-9faf-68b71dc1b673_calibrate_1101475_2.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/075b50ef-9076-43c9-a6f7-f56866eb0314_characterizeImage_1101475_2.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/075b50ef-9076-43c9-a6f7-f56866eb0314_characterizeImage_1101475_2.stderr parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b12b3c34f4f1f5ba0ba591657876544d with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash dc013aa10e13bbd08cfbe7970259db69 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/b7d09354-ade8-450e-bd88-832a1601b503_calibrate_1101469_6.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/b7d09354-ade8-450e-bd88-832a1601b503_calibrate_1101469_6.stderr parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c06c6598-337d-4f2f-b32e-5ed1b7d9cdfe_characterizeImage_1101469_6.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/c06c6598-337d-4f2f-b32e-5ed1b7d9cdfe_characterizeImage_1101469_6.stderr parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0674deb369fe62c91b1bfb957566897b with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 6a35189971549351656b5530d01dc940 parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a05120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/38656884-d96c-4535-a88d-150fb5b06457_calibrate_1101475_56.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/38656884-d96c-4535-a88d-150fb5b06457_calibrate_1101475_56.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c132514c-76fb-4e82-97b7-bbc23d0aafe3_characterizeImage_1101475_56.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c132514c-76fb-4e82-97b7-bbc23d0aafe3_characterizeImage_1101475_56.stderr parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 726f46e0cb292d58bbb99740f388c258 with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash f2fdbd756d39879668e64d8f5468b35e parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/68f591f8-9859-4bb4-9253-05dc3e47b577_calibrate_1101466_22.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/68f591f8-9859-4bb4-9253-05dc3e47b577_calibrate_1101466_22.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/040b5c50-d5fa-46b9-92e3-65952c61ea15_characterizeImage_1101466_22.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/040b5c50-d5fa-46b9-92e3-65952c61ea15_characterizeImage_1101466_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:47 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74f8056d8a1bb802c70599beaf29b218 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 8368b439c0ccf41e98b00095ae2583ed parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/ded1ceb7-2dc8-464e-8a28-e439abc22e6c_calibrate_1101475_10.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/ded1ceb7-2dc8-464e-8a28-e439abc22e6c_calibrate_1101475_10.stderr parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/7b86f3de-b0d4-4a77-aa93-525453f557af_characterizeImage_1101475_10.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/7b86f3de-b0d4-4a77-aa93-525453f557af_characterizeImage_1101475_10.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02562d0ab739e693af35b8c06a1eddf7 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/7accc0e7-bbe9-48a9-bf7e-794b3cb34e28_calibrate_1101472_31.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/7accc0e7-bbe9-48a9-bf7e-794b3cb34e28_calibrate_1101472_31.stderr parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b62d86f38647429a660bcc4b7199479 with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash c6271864bb231b08a61118201409262f parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2a963dc7-1ca2-4ad5-9075-74f3c78e8de8_calibrate_1101466_47.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2a963dc7-1ca2-4ad5-9075-74f3c78e8de8_calibrate_1101466_47.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/44a9e101-8901-416e-a0d9-6bdb1e5347f2_characterizeImage_1101466_47.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/44a9e101-8901-416e-a0d9-6bdb1e5347f2_characterizeImage_1101466_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:48 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 708af50befa8ba787158e23c37d2ab5b with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/26d87726-3962-4b78-acb3-b8690c3413ab_calibrate_1101466_30.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/26d87726-3962-4b78-acb3-b8690c3413ab_calibrate_1101466_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:50 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5f27e3125af0805b52b7622181fdfba with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash 44688bec43125e0f34149ecb9288f896 parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a07100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/1b0701a4-e7cd-4ed4-8530-a5a826e83fe9_calibrate_1101475_23.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/1b0701a4-e7cd-4ed4-8530-a5a826e83fe9_calibrate_1101475_23.stderr parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/9aaf079c-9de7-4c31-a17e-0c01c4e03767_characterizeImage_1101475_23.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/9aaf079c-9de7-4c31-a17e-0c01c4e03767_characterizeImage_1101475_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:52 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9f9e39a58b59764fb4f2b4e5591cd9a with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7a147a2d-7203-4bf9-9619-8f2d405dad09_calibrate_1101469_47.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7a147a2d-7203-4bf9-9619-8f2d405dad09_calibrate_1101469_47.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0ba5f4490eb3175ca40d45d065835d7 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/97cc0b8e-97e6-4128-ba0d-9ce7dfd751a0_calibrate_1101466_34.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/97cc0b8e-97e6-4128-ba0d-9ce7dfd751a0_calibrate_1101466_34.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65f8cab3a4db4f7b7afa738131e7f9dc with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 02e9b22efdc1ea58fd0c999ecab600de parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/b14025d5-64b2-462c-aa38-0a3da52649c1_calibrate_1101472_48.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/b14025d5-64b2-462c-aa38-0a3da52649c1_calibrate_1101472_48.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/6621a70a-d337-4c77-aec1-33ea1c758628_characterizeImage_1101472_48.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/6621a70a-d337-4c77-aec1-33ea1c758628_characterizeImage_1101472_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:53 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46398e283ab7328883a29c3c0997d678 with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash 4b14523d6b351cd73e5eb8e59eaae3d2 parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b74c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/d9591c1e-2d16-4747-8021-c31adfb4c5b9_calibrate_1101466_4.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/d9591c1e-2d16-4747-8021-c31adfb4c5b9_calibrate_1101466_4.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/90eaed79-86cb-405d-a71f-791c7790c04e_characterizeImage_1101466_4.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/90eaed79-86cb-405d-a71f-791c7790c04e_characterizeImage_1101466_4.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0960b7d79e7f1a338f9b2816e2ce055c with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1cd731e0-e740-4787-b090-47d96fcd2b16_calibrate_1101466_14.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1cd731e0-e740-4787-b090-47d96fcd2b16_calibrate_1101466_14.stderr parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 261987e7de076c5b3aeaf5f9efc82480 with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash b8b4ee4843ab2852ac0cfc75f2cf9aff parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/40ac9ba0-c6da-444a-8f3a-284455726bce_calibrate_1101466_43.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/40ac9ba0-c6da-444a-8f3a-284455726bce_calibrate_1101466_43.stderr parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/f85bd32a-edcc-45ae-8efc-6dae6b247d9c_characterizeImage_1101466_43.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/f85bd32a-edcc-45ae-8efc-6dae6b247d9c_characterizeImage_1101466_43.stderr parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b714184d1fc8d48d76f6feb89de7ba69 with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 073d0c9bc55deb33dfaa134f2e4ffc48 parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/4cead598-7177-47ca-8d06-97e04668f9b4_calibrate_1101472_25.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/4cead598-7177-47ca-8d06-97e04668f9b4_calibrate_1101472_25.stderr parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/62830edd-3a21-4165-8fda-50d4a4f85b5a_characterizeImage_1101472_25.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/62830edd-3a21-4165-8fda-50d4a4f85b5a_characterizeImage_1101472_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:55 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry addd529ac0d62b4529563da48ac47703 with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash 792bdb65ad3da0d167e4c89833c6f7e4 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b77100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/fab958d8-247c-4350-808a-7de4a4f48faa_calibrate_1101469_59.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/fab958d8-247c-4350-808a-7de4a4f48faa_calibrate_1101469_59.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/ae21c014-63dc-46fd-bc2d-e3fd3c08852d_characterizeImage_1101469_59.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/ae21c014-63dc-46fd-bc2d-e3fd3c08852d_characterizeImage_1101469_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:57 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d2380f0cf884289814966126dc048c3 with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 64a89f61046da88bad7b5fd637626bb7 parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5cc2b083-0d73-47b4-8041-56db75780051_calibrate_1101466_58.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5cc2b083-0d73-47b4-8041-56db75780051_calibrate_1101466_58.stderr parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/f0d1938e-ce4b-492e-bdfa-89f26dc3ab03_characterizeImage_1101466_58.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/f0d1938e-ce4b-492e-bdfa-89f26dc3ab03_characterizeImage_1101466_58.stderr parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57220570230170300cf8bee13a9ea07a with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash 392621426fc9662a91fadac080be421d parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/108bdce6-133f-40b9-b1ff-723bcd72dc87_calibrate_1101472_10.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/108bdce6-133f-40b9-b1ff-723bcd72dc87_calibrate_1101472_10.stderr parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/02e9c2e7-f7d7-4571-803e-6cf3dbfbc386_characterizeImage_1101472_10.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/02e9c2e7-f7d7-4571-803e-6cf3dbfbc386_characterizeImage_1101472_10.stderr parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b700c664a90bf0b1ee5c70b31aaa1c55 with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 90eff0c1118bb4543899373cab9f67eb parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d2791c23-310d-4110-a0b8-ee462bf12901_calibrate_1101475_4.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d2791c23-310d-4110-a0b8-ee462bf12901_calibrate_1101475_4.stderr parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a50a4147-fe8e-4a7e-96ae-7135836c071e_characterizeImage_1101475_4.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a50a4147-fe8e-4a7e-96ae-7135836c071e_characterizeImage_1101475_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:06:58 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bed7ad28daa74ee3280e3a63e965f98 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/3e1a0a67-edde-47aa-8643-168b1e9525a5_calibrate_1101469_50.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/3e1a0a67-edde-47aa-8643-168b1e9525a5_calibrate_1101469_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:00 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa4cf43e79e37cf5f8229746143f6f84 with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 8f3ecfa387ff108eb039997730debb35 parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ec220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/33e3bd39-9466-45c8-b130-9f77c2eb23f0_calibrate_1101469_29.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/33e3bd39-9466-45c8-b130-9f77c2eb23f0_calibrate_1101469_29.stderr parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/baa97767-9413-4ae0-86fb-9fe5ea018ab1_characterizeImage_1101469_29.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/baa97767-9413-4ae0-86fb-9fe5ea018ab1_characterizeImage_1101469_29.stderr parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6ad6f92888e42143011df4b7c04c296 with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 5377e2be11a0cde1812e67cdcd58e35c parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b75e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/a10b4be4-75ac-4dd3-a718-a64ca8a10ced_calibrate_1101472_46.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/a10b4be4-75ac-4dd3-a718-a64ca8a10ced_calibrate_1101472_46.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/5a07453c-101a-4b27-9f0a-f3fc7187456f_characterizeImage_1101472_46.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/5a07453c-101a-4b27-9f0a-f3fc7187456f_characterizeImage_1101472_46.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa0f23c17fd5ccd996125f9aff962ba1 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/3da22e3a-763b-4943-9d2c-274d04c8e38e_calibrate_1101469_40.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/3da22e3a-763b-4943-9d2c-274d04c8e38e_calibrate_1101469_40.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd8661637b097f7d20eb83d6e5030f60 with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash c957a818674d20d65b66945d9e65095a parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b76160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5df76fd5-7062-4fac-b4c1-0274831a2b20_calibrate_1101466_26.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5df76fd5-7062-4fac-b4c1-0274831a2b20_calibrate_1101466_26.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/bc7f7478-6e0b-4aa1-a707-14e75a17a97f_characterizeImage_1101466_26.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/bc7f7478-6e0b-4aa1-a707-14e75a17a97f_characterizeImage_1101466_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:02 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6041e50821597dc7301039ca09fa50d with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash fdf654dcfe1afcac289c16f8718b9b6a parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b3b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/16c85dff-7257-4674-a02f-232b713af242_calibrate_1101466_21.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/16c85dff-7257-4674-a02f-232b713af242_calibrate_1101466_21.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/750275a7-c013-4afa-8695-80fa3364cb64_characterizeImage_1101466_21.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/750275a7-c013-4afa-8695-80fa3364cb64_characterizeImage_1101466_21.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79a2c4027ee1e783bd10fdb1463544b3 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1d94e71e-de00-41cb-af8b-8e33c2673260_calibrate_1101466_1.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1d94e71e-de00-41cb-af8b-8e33c2673260_calibrate_1101466_1.stderr parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97106baa6d443e2d82adbb3f5d87b59b with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash dc7b2d379c5e519dd5279744cacf289b parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/64bc9fe1-2992-4309-a53a-46e450ecde4a_calibrate_1101472_34.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/64bc9fe1-2992-4309-a53a-46e450ecde4a_calibrate_1101472_34.stderr parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/63e73a4e-50fb-41c8-a9cf-e10f6a4dea2d_characterizeImage_1101472_34.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/63e73a4e-50fb-41c8-a9cf-e10f6a4dea2d_characterizeImage_1101472_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:04 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7611e6edfccceec6afcc557e3bf5d4be with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/021c9f26-f89f-4a0d-8d3f-6464855e1737_calibrate_1101469_38.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/021c9f26-f89f-4a0d-8d3f-6464855e1737_calibrate_1101469_38.stderr parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 561be792900fdc6db8b3c71a7bfc4060 with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 490ac97dd69030b0c40c3031cc499f3b parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ec860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ccdc191c-cc2c-4f46-8e7a-aa539d31dd04_calibrate_1101472_45.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ccdc191c-cc2c-4f46-8e7a-aa539d31dd04_calibrate_1101472_45.stderr parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/c0f8ad3f-91cb-432f-89d3-6fbe962e341a_characterizeImage_1101472_45.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/c0f8ad3f-91cb-432f-89d3-6fbe962e341a_characterizeImage_1101472_45.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce2835d76eb3f00893a1d05499957233 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/82f820ea-31f3-4252-a6ed-8bc035e95e69_calibrate_1101466_27.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/82f820ea-31f3-4252-a6ed-8bc035e95e69_calibrate_1101466_27.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a2232ff55a97e10018e96dbcf65e2ba with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2cddefd4-5bfb-43b1-8888-34958b2ea148_calibrate_1101469_57.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2cddefd4-5bfb-43b1-8888-34958b2ea148_calibrate_1101469_57.stderr parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c30bba3bf24d6d6787d87772450e201 with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 4ee62c2f8d847bd525e95f2d72c759f5 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/f0d872a2-5f9c-4e08-ae2f-e400097f6aed_calibrate_1101469_21.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/f0d872a2-5f9c-4e08-ae2f-e400097f6aed_calibrate_1101469_21.stderr parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/46a4e94f-a547-4de9-b9db-bb01375dedf9_characterizeImage_1101469_21.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/46a4e94f-a547-4de9-b9db-bb01375dedf9_characterizeImage_1101469_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:06 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d62f8fca5b658d53ff25a15ce1880a6 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/90ca7623-bc21-495e-8839-c3f8ac7d2157_calibrate_1101469_32.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/90ca7623-bc21-495e-8839-c3f8ac7d2157_calibrate_1101469_32.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bffce22ad8b8a3954476d3273477dcb8 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/aa613804-4eb1-4b49-ab6a-314e5c855974_calibrate_1101469_14.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/aa613804-4eb1-4b49-ab6a-314e5c855974_calibrate_1101469_14.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 121ec43af6f8afae07c15aa677eecbad with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/21128926-d2f1-422c-b3d8-d4eabdc7300e_calibrate_1101466_57.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/21128926-d2f1-422c-b3d8-d4eabdc7300e_calibrate_1101466_57.stderr parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d512f3b1a164efe722909651ef9447c with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash ba32d62ca16a58aac9101aa4cd8f805b parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b75580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/13b1a916-fe38-4f2e-973b-856dba471436_calibrate_1101469_22.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/13b1a916-fe38-4f2e-973b-856dba471436_calibrate_1101469_22.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/64c6b6e4-3eb0-4dae-a16c-11ff19fe6a87_characterizeImage_1101469_22.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/64c6b6e4-3eb0-4dae-a16c-11ff19fe6a87_characterizeImage_1101469_22.stderr parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bacea6909e9dc4325c6bd819b698d734 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 61a6eaca5890806fba7cb20447f0fea9 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/38113d78-bba4-492d-bf5b-6cacb3e374b2_calibrate_1101475_3.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/38113d78-bba4-492d-bf5b-6cacb3e374b2_calibrate_1101475_3.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/0101d367-2781-466f-8608-bc09e91e4fb8_characterizeImage_1101475_3.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/0101d367-2781-466f-8608-bc09e91e4fb8_characterizeImage_1101475_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:08 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 143883b13b47be2b2b5bca18f51bb14d with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 476fa07b676eb13bb11b49cb331b805b parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4ad4e69a-b8de-427a-830e-98de5dbbe899_calibrate_1101466_62.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4ad4e69a-b8de-427a-830e-98de5dbbe899_calibrate_1101466_62.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e93fae63-34a6-4939-9360-cc7dd287b2cf_characterizeImage_1101466_62.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/e93fae63-34a6-4939-9360-cc7dd287b2cf_characterizeImage_1101466_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:09 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29433438bb2f0774455958ef370a00cc with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash d9b61258abcaf6dac028f8105a0852e0 parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ef7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/6f32275c-fec4-4c98-b7f9-72049872c6ae_calibrate_1101475_55.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/6f32275c-fec4-4c98-b7f9-72049872c6ae_calibrate_1101475_55.stderr parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/930a545f-e962-44cd-9a84-e7005d50a790_characterizeImage_1101475_55.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/930a545f-e962-44cd-9a84-e7005d50a790_characterizeImage_1101475_55.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f341860d28ea4720cc15e28911f4678c with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/bb9f3b50-eb8d-4528-9445-0334688b9f87_calibrate_1101475_31.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/bb9f3b50-eb8d-4528-9445-0334688b9f87_calibrate_1101475_31.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d5cbb1ebc6ea2ea1085ad7cd917403f with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash 80a5c25be3469cf891cb2eb6fd6e8e33 parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/1df71d75-fa80-4d12-97a6-47803efc1bef_calibrate_1101469_26.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/1df71d75-fa80-4d12-97a6-47803efc1bef_calibrate_1101469_26.stderr parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/3096b2cc-96ee-4c23-970b-c3927271b530_characterizeImage_1101469_26.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/3096b2cc-96ee-4c23-970b-c3927271b530_characterizeImage_1101469_26.stderr parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e9629d88b769c253b4c005db74158fc with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash a5b4f673a50cf16dc969bea133c7c5a2 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/07a1240f-db2c-4cf6-bb3c-1cbc02c65f7d_calibrate_1101466_29.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/07a1240f-db2c-4cf6-bb3c-1cbc02c65f7d_calibrate_1101466_29.stderr parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/2db12c8b-c956-4a61-a5bc-ff20694be85d_characterizeImage_1101466_29.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/2db12c8b-c956-4a61-a5bc-ff20694be85d_characterizeImage_1101466_29.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8ced5c98d673d63aeee8d8ace3bb4fc with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4e217ba9-8757-4008-81d2-fa5031576dd5_calibrate_1101466_60.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4e217ba9-8757-4008-81d2-fa5031576dd5_calibrate_1101466_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:11 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e5d31579c9d77ee4ad4b08b6365104c with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/447aaaf6-245e-4a26-b11c-4eb033b834f8_calibrate_1101469_15.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/447aaaf6-245e-4a26-b11c-4eb033b834f8_calibrate_1101469_15.stderr parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99e89c560dc7150cc8ca22e7b1174d61 with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash bf6e7fcbd8446257f944d761e7a00183 parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608ed120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/999118cc-5056-4195-8757-83936856b586_calibrate_1101475_34.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/999118cc-5056-4195-8757-83936856b586_calibrate_1101475_34.stderr parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/33553aa4-3b02-4610-9b80-4f1e3543fe10_characterizeImage_1101475_34.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/33553aa4-3b02-4610-9b80-4f1e3543fe10_characterizeImage_1101475_34.stderr parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a04e6ef6671d41beddca466c81d4317 with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 99ad34e249d32538f48b6c9d3d5ebb57 parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/dccaf65e-741a-453f-b2ac-ba4fbade8aa6_calibrate_1101466_33.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/dccaf65e-741a-453f-b2ac-ba4fbade8aa6_calibrate_1101466_33.stderr parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/dc406187-6604-4929-a509-8f85f13a8794_characterizeImage_1101466_33.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/dc406187-6604-4929-a509-8f85f13a8794_characterizeImage_1101466_33.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c7032a2c496d24b3dc7938151ddf4b9 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5ec2190b-d156-4638-9fc3-605182f7842e_calibrate_1101466_18.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5ec2190b-d156-4638-9fc3-605182f7842e_calibrate_1101466_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:13 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d620bb2a45dded8bd0990f92c6d066c1 with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 47829c94d159cf4b87eb314fe48eef68 parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/37be4d4c-0efd-474a-aa48-3a5985bd07af_calibrate_1101469_16.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/37be4d4c-0efd-474a-aa48-3a5985bd07af_calibrate_1101469_16.stderr parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/66a6216b-a828-490c-9335-176ad9cfcfca_characterizeImage_1101469_16.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/66a6216b-a828-490c-9335-176ad9cfcfca_characterizeImage_1101469_16.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20959bd011166cefd1d8d98df727eb47 with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash c580ca00c7a9da63b3998bdddab46bb1 parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b41a1686-b64d-4c96-9585-8ca3cb15917f_calibrate_1101466_45.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b41a1686-b64d-4c96-9585-8ca3cb15917f_calibrate_1101466_45.stderr parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/ba710486-f66c-47f3-b5ae-2d6a0ce6a01b_characterizeImage_1101466_45.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/ba710486-f66c-47f3-b5ae-2d6a0ce6a01b_characterizeImage_1101466_45.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78ec10bcd83b09837056e8a72bf613a6 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 13a0f9e71af4a968c0983df856f7c968 parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b77c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/e6e89b66-001d-46bd-a0ab-ba944a180502_calibrate_1101469_52.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/e6e89b66-001d-46bd-a0ab-ba944a180502_calibrate_1101469_52.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/f83e75c4-ed4e-4297-a10e-3fe154830593_characterizeImage_1101469_52.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/f83e75c4-ed4e-4297-a10e-3fe154830593_characterizeImage_1101469_52.stderr parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1567b81f8cc1468f71e6c303437c6636 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash 91228a1baef699aeb39676fd74986654 parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a047c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/bddeaad2-c731-4119-9fce-debc6c8a046a_calibrate_1101475_11.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/bddeaad2-c731-4119-9fce-debc6c8a046a_calibrate_1101475_11.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c0ea04aa-a9ef-4f3f-beae-feee370cb26d_characterizeImage_1101475_11.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/c0ea04aa-a9ef-4f3f-beae-feee370cb26d_characterizeImage_1101475_11.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 996eb8c6c7f9c9a612701395a991ea98 with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash 0123948d78a9eafec3894073f4b6d21e parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d67c80b8-e0d6-4962-9305-74bc13d414fb_calibrate_1101472_53.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d67c80b8-e0d6-4962-9305-74bc13d414fb_calibrate_1101472_53.stderr parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/769c713d-11cb-4d27-be90-7d91309a12e6_characterizeImage_1101472_53.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/769c713d-11cb-4d27-be90-7d91309a12e6_characterizeImage_1101472_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:15 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 610 try 1 failed parsl.dataflow.dflow ERROR: Task 610 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7c59eba3cd2f6e901c8bfee58ac0649a with result from task 610 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 611 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 611 try 0 failed parsl.dataflow.dflow INFO: Task 611 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/2d518259-7895-40f2-849b-d25eb66c675a_calibrate_1101472_2.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/2d518259-7895-40f2-849b-d25eb66c675a_calibrate_1101472_2.stderr parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8f528bae-823c-4669-a83f-423304c44523_characterizeImage_1101472_2.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/8f528bae-823c-4669-a83f-423304c44523_characterizeImage_1101472_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:16 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd2fe7679a8ef412355d5382db617f82 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b70b3a48-e067-485d-a8d4-70537f635d82_calibrate_1101466_12.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b70b3a48-e067-485d-a8d4-70537f635d82_calibrate_1101466_12.stderr parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 258326f4befeedbea04f82c3ece7d661 with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash c10e67afd2950584602f873eb0c78f7f parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8510e8eb-08cd-4199-8db4-cfc39c59c304_calibrate_1101469_28.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8510e8eb-08cd-4199-8db4-cfc39c59c304_calibrate_1101469_28.stderr parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/b9faa86c-a60f-4f81-854f-e0c53abe1d96_characterizeImage_1101469_28.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/b9faa86c-a60f-4f81-854f-e0c53abe1d96_characterizeImage_1101469_28.stderr parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5900145034421ee88bee60e25b01f1e1 with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 212156acde0a657425c38e6257b8b22d parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c7e309f4-2090-4919-b6e6-3d871312d099_calibrate_1101475_46.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c7e309f4-2090-4919-b6e6-3d871312d099_calibrate_1101475_46.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/283b002b-8244-47bf-9aa8-ca08b51c929d_characterizeImage_1101475_46.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/283b002b-8244-47bf-9aa8-ca08b51c929d_characterizeImage_1101475_46.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f5ed16b9001d34b5561a0622bb3b22c with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/cddbe314-394e-4a51-8051-976d1a7d2398_calibrate_1101469_7.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/cddbe314-394e-4a51-8051-976d1a7d2398_calibrate_1101469_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:17 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a43091f6b9a55e59054506fee7a70a97 with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 55a020d20d83c4158ac9baf5efbb2b46 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b77920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/1b6e3141-7003-4d1c-add6-236d8e2881d4_calibrate_1101472_38.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/1b6e3141-7003-4d1c-add6-236d8e2881d4_calibrate_1101472_38.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/aae529ce-4359-4d6a-9abf-67e46e949fe2_characterizeImage_1101472_38.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/aae529ce-4359-4d6a-9abf-67e46e949fe2_characterizeImage_1101472_38.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5339f5e31493474d14b08c838365687 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 005c6993fe55dddbfa544ead6e03be9e parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8ef405d5-72f3-472d-9263-e0e5943bd894_calibrate_1101469_56.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8ef405d5-72f3-472d-9263-e0e5943bd894_calibrate_1101469_56.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2e9aa646-860d-45ee-b987-a0bd13dd6b01_characterizeImage_1101469_56.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/2e9aa646-860d-45ee-b987-a0bd13dd6b01_characterizeImage_1101469_56.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22b3e9a98a18e0cfc91ec36822c7c082 with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 9d38a22fa8aadc0c890971ef5c08c52c parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a06d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d625ea1f-bbb4-4e0a-881c-d7765eaac42b_calibrate_1101472_21.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d625ea1f-bbb4-4e0a-881c-d7765eaac42b_calibrate_1101472_21.stderr parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d819302c-3880-463e-a951-536454af31a3_characterizeImage_1101472_21.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/d819302c-3880-463e-a951-536454af31a3_characterizeImage_1101472_21.stderr parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a79291baacada01d90943f7f272f18f9 with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 79cc997e5a021652df4783ae5f3eea29 parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/7c4eae54-2f04-43a9-bee7-d2cbd6019f2a_calibrate_1101472_37.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/7c4eae54-2f04-43a9-bee7-d2cbd6019f2a_calibrate_1101472_37.stderr parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/3c707758-39b6-4827-99c6-6af94e8b8957_characterizeImage_1101472_37.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/3c707758-39b6-4827-99c6-6af94e8b8957_characterizeImage_1101472_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:19 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e6abd2ce6f1643786b5a98f40bb3c1e with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/ee4a175f-eb03-40e4-aec1-4682b38f327a_calibrate_1101466_6.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/ee4a175f-eb03-40e4-aec1-4682b38f327a_calibrate_1101466_6.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93d847fffa4db9411df6eac2eb907675 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/00d87b46-32a5-4503-9747-6891c03a8e03_calibrate_1101466_16.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/00d87b46-32a5-4503-9747-6891c03a8e03_calibrate_1101466_16.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4774ca7028248e41f8b22049482f2ff with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/881f5d21-c888-4f88-a548-927b6c8ef6b3_calibrate_1101469_5.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/881f5d21-c888-4f88-a548-927b6c8ef6b3_calibrate_1101469_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:21 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00cf3b86f68d6b83847ec06074ba610f with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/71a11247-94cb-4b1b-adae-e20dbadde565_calibrate_1101469_51.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/71a11247-94cb-4b1b-adae-e20dbadde565_calibrate_1101469_51.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1002281313920bbc7daaebc38573077a with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/e947d18d-261c-4520-b663-e01e9cf1c742_calibrate_1101469_27.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/e947d18d-261c-4520-b663-e01e9cf1c742_calibrate_1101469_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:22 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55f3b7d9fb274af9a72ba6b0e7377681 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/dc0170ee-77fa-46b8-b950-e8d7b4092574_calibrate_1101469_34.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/dc0170ee-77fa-46b8-b950-e8d7b4092574_calibrate_1101469_34.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 017f77177806ceb7d453c5700c1a93f1 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ecc09f7b-7e34-44b5-8e22-5dd8f23dde49_calibrate_1101472_24.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ecc09f7b-7e34-44b5-8e22-5dd8f23dde49_calibrate_1101472_24.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ed8cc3039c07336c37fd67c15a08774 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/479ea0e9-a820-4aa8-b92f-78b0c3207c05_calibrate_1101472_40.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/479ea0e9-a820-4aa8-b92f-78b0c3207c05_calibrate_1101472_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:24 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af125c4fd85b0e073329914993e37af2 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1873e2c7-6e55-4284-8ace-6620d2d44163_calibrate_1101466_17.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1873e2c7-6e55-4284-8ace-6620d2d44163_calibrate_1101466_17.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1765fd105dec064b98020b7d7c7622e1 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ba186166-dd24-4b03-8db5-b0256c4a9007_calibrate_1101472_49.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ba186166-dd24-4b03-8db5-b0256c4a9007_calibrate_1101472_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 213 parsl.dataflow.strategy DEBUG: Executor multi has 213 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:07:26 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:07:26 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7e48b184bdcaebd9db272b12fff67d1 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/eed18bc9-60f5-43da-836d-d4323fdd766e_calibrate_1101469_13.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/eed18bc9-60f5-43da-836d-d4323fdd766e_calibrate_1101469_13.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c71ec6ce35c2090268813f7378d25df with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/41664c2a-5762-4d5d-9a6f-fbfb806431eb_calibrate_1101469_33.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/41664c2a-5762-4d5d-9a6f-fbfb806431eb_calibrate_1101469_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:27 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f36c4e0efcdee72a1457bfb90b65c6a5 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/c6f4de20-b76c-41d3-83de-e2c267652969_calibrate_1101472_18.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/c6f4de20-b76c-41d3-83de-e2c267652969_calibrate_1101472_18.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8f27b0f6d2ecd3e091fcd4ec4836074 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/8df4d422-3b3b-4687-b894-86d067d6d9b8_calibrate_1101475_24.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/8df4d422-3b3b-4687-b894-86d067d6d9b8_calibrate_1101475_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:29 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a55b27297bfabafc186939301052aff with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/a31416da-5b5b-49a5-8d03-1d845bb1d1b6_calibrate_1101469_8.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/a31416da-5b5b-49a5-8d03-1d845bb1d1b6_calibrate_1101469_8.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 159fa80b5ea390b685f3826db0e87b4e with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/76b0b623-6813-4147-820b-fbf6ed440d0f_calibrate_1101472_52.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/76b0b623-6813-4147-820b-fbf6ed440d0f_calibrate_1101472_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:31 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.memoization DEBUG: Storing app cache entry 5c81d08ac8cab9b82495df565a386297 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/4bc6c48c-e631-4b93-ac9f-ae6cbd084f9f_calibrate_1101469_11.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/4bc6c48c-e631-4b93-ac9f-ae6cbd084f9f_calibrate_1101469_11.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 018de0128233e74bffb7812dc4913dc1 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/8f7210c8-4994-4c0d-90e3-1e13e8083692_calibrate_1101472_51.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/8f7210c8-4994-4c0d-90e3-1e13e8083692_calibrate_1101472_51.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6eab96d9128e379bdccad341e4637c9 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/85639a6f-31d4-4058-bfc6-169078579509_calibrate_1101466_19.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/85639a6f-31d4-4058-bfc6-169078579509_calibrate_1101466_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:33 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 473c22008597eeb5bae037b1caecce58 with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash d7bcbbcc46c66b8f6bbcdb85ea5e91a8 parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b77ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4726763a-b245-41f0-9da1-3a509af4c58e_calibrate_1101466_48.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4726763a-b245-41f0-9da1-3a509af4c58e_calibrate_1101466_48.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c968575d-6bec-40ec-8c9c-41316090f190_characterizeImage_1101466_48.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/c968575d-6bec-40ec-8c9c-41316090f190_characterizeImage_1101466_48.stderr parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 612c445a8a9bc9a880522fb6ad93ea72 with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash e222e888de699e4df29871a5c01d9093 parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461608efec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/02db57bc-3e56-4458-ba24-fa1232bd3f34_calibrate_1101475_42.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/02db57bc-3e56-4458-ba24-fa1232bd3f34_calibrate_1101475_42.stderr parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8eaf6abe-dd4e-4997-a835-5550655038d8_characterizeImage_1101475_42.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/8eaf6abe-dd4e-4997-a835-5550655038d8_characterizeImage_1101475_42.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e869800397392951c1c4ac357527eacf with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/e591665f-8b01-4ed2-969f-10b3408241d6_calibrate_1101475_53.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/e591665f-8b01-4ed2-969f-10b3408241d6_calibrate_1101475_53.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bda879fa5ccbaa1456386304ba56a419 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/a7ffd712-e288-4b5f-a753-aa64815cdd60_calibrate_1101469_20.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/a7ffd712-e288-4b5f-a753-aa64815cdd60_calibrate_1101469_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:35 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47814b1df7108c88ce7fcb39d6cece94 with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash e007fd96b8a05e6c594a57aadd243c14 parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b74f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/803ddc78-9b88-402b-9831-7ee3b8c454fe_calibrate_1101472_29.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/803ddc78-9b88-402b-9831-7ee3b8c454fe_calibrate_1101472_29.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/85d52d3b-34f4-403b-b2e9-4842ab9f066d_characterizeImage_1101472_29.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/85d52d3b-34f4-403b-b2e9-4842ab9f066d_characterizeImage_1101472_29.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0b83639dd6c17fba88ee0cbd39bf566 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ec1356fd-7d8c-4d93-9588-2f240889cd75_calibrate_1101472_39.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ec1356fd-7d8c-4d93-9588-2f240889cd75_calibrate_1101472_39.stderr parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2ebb0b02c90f55d541642fa70c83ccb with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash b6903257f78cf6012ef9e7878ee3e99a parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ba5434e9-5790-454f-aa4b-c67ea309a4d9_calibrate_1101472_44.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ba5434e9-5790-454f-aa4b-c67ea309a4d9_calibrate_1101472_44.stderr parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4ea3ad58-ffc5-4fce-9cb3-8cd8654c7995_characterizeImage_1101472_44.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/4ea3ad58-ffc5-4fce-9cb3-8cd8654c7995_characterizeImage_1101472_44.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 581627e1ed9d70f74366ea14b093a772 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/1a54754e-b0f3-4ddf-b3ca-c04574ebd459_calibrate_1101475_52.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/1a54754e-b0f3-4ddf-b3ca-c04574ebd459_calibrate_1101475_52.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9277ba2f6e00b6b52e6b7680b9b7e66 with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash fd2ff24555c76af8bb2345a59bf92c55 parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b74540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/77bb4ae2-130c-4198-b092-c64604310dc3_calibrate_1101472_59.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/77bb4ae2-130c-4198-b092-c64604310dc3_calibrate_1101472_59.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/70db0906-bcc8-461b-8c59-b0f6257e9ea8_characterizeImage_1101472_59.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/70db0906-bcc8-461b-8c59-b0f6257e9ea8_characterizeImage_1101472_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:36 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1effb4a31340c7009b541b05eb12bfbb with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/2bc10262-bb8c-41f0-942a-75349a5a449f_calibrate_1101472_56.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/2bc10262-bb8c-41f0-942a-75349a5a449f_calibrate_1101472_56.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59c8996421a58ae9101ddccaab2b027e with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash 03867228ac49e5e0eda4e010649fe943 parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5ff230f8-3735-447e-b9ce-51deb4ce4f99_calibrate_1101475_30.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5ff230f8-3735-447e-b9ce-51deb4ce4f99_calibrate_1101475_30.stderr parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/088bbc04-438b-40e2-8ec0-2e129cb0a1dd_characterizeImage_1101475_30.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/088bbc04-438b-40e2-8ec0-2e129cb0a1dd_characterizeImage_1101475_30.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc1d8f211de6175ab6b8edd28531bb6a with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/c5eed667-a636-468a-9584-8d8fa7639fbc_calibrate_1101466_31.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/c5eed667-a636-468a-9584-8d8fa7639fbc_calibrate_1101466_31.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64cade800efbb20739cb4e3b05bb6a58 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/1de05074-3a3c-42b3-830b-0184fddb113f_calibrate_1101475_51.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/1de05074-3a3c-42b3-830b-0184fddb113f_calibrate_1101475_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:38 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1897ceccb6701d80f6ca3b60ab34beb with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/6f4717b6-0f2e-4aca-af75-e87b18909b17_calibrate_1101472_15.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/6f4717b6-0f2e-4aca-af75-e87b18909b17_calibrate_1101472_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:40 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:42 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c317d866f2e48b2400d8b5ebe01c47c with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 9781452c8765430f666c557de5005f04 parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b758a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/0ef2dbd1-956a-4f7b-a870-c27160dd0808_calibrate_1101475_33.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/0ef2dbd1-956a-4f7b-a870-c27160dd0808_calibrate_1101475_33.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/037ac4e7-ddad-4bfb-bb20-ef9a0852e338_characterizeImage_1101475_33.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/037ac4e7-ddad-4bfb-bb20-ef9a0852e338_characterizeImage_1101475_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:44 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ac934a1474c6dab12037277a6047b06 with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash 0da14707bd6fbe77436bbb0596022651 parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7a9aef70-38b0-4c05-bcb1-8357804bd189_calibrate_1101469_36.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7a9aef70-38b0-4c05-bcb1-8357804bd189_calibrate_1101469_36.stderr parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/f79e9085-e809-412a-8cc9-eaaf3a9c4b0a_characterizeImage_1101469_36.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/f79e9085-e809-412a-8cc9-eaaf3a9c4b0a_characterizeImage_1101469_36.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f475a85bfae030f99522aa9a8379a3ca with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d9bdab03-ad31-407f-bcca-cee8145c8226_calibrate_1101472_58.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d9bdab03-ad31-407f-bcca-cee8145c8226_calibrate_1101472_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:46 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f2fac09bdc25e1b02999583720433a0 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5fb79185-c4bc-4c80-b04e-be24be4190f1_calibrate_1101475_25.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5fb79185-c4bc-4c80-b04e-be24be4190f1_calibrate_1101475_25.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 520edbe152bce2278a6f0033f6fa69d1 with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/db199cff-394e-4540-9a44-96ecfa62df1d_calibrate_1101469_19.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/db199cff-394e-4540-9a44-96ecfa62df1d_calibrate_1101469_19.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f325723523cc9e1b97a47a7dcbdac6f4 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/ca1a5a30-dd1d-45c8-924d-37998e9e8aac_calibrate_1101466_49.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/ca1a5a30-dd1d-45c8-924d-37998e9e8aac_calibrate_1101466_49.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95effa69c0ff7487bfaf90f5fd6980d9 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/f289ca8d-2db7-416a-82d9-5f1d3ce804ba_calibrate_1101475_27.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/f289ca8d-2db7-416a-82d9-5f1d3ce804ba_calibrate_1101475_27.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f8ff938eb24a46ad4bbd2af41581529 with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash fb5315387290f6d54734e8a365d2b9fc parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b75940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/981cd3f4-bb85-4110-9604-2375ce760899_calibrate_1101472_60.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/981cd3f4-bb85-4110-9604-2375ce760899_calibrate_1101472_60.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/2ff0b43a-caea-4b48-9215-30260cfdf2b6_characterizeImage_1101472_60.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/2ff0b43a-caea-4b48-9215-30260cfdf2b6_characterizeImage_1101472_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:47 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba48374ac064c6ec4da8a57b8b0dd91f with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/49e33f19-90e9-49fe-815e-eb3db7ae72e5_calibrate_1101469_17.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/49e33f19-90e9-49fe-815e-eb3db7ae72e5_calibrate_1101469_17.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2a44bfd974c379c51c89dcf08ac8fdc with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5ea4825b-8eb8-4fea-8b83-3283c34d34e2_calibrate_1101466_52.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5ea4825b-8eb8-4fea-8b83-3283c34d34e2_calibrate_1101466_52.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f0ae33de9d10bc3b57c921234e1aebb with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1bbf4df7-aaf3-4d5b-8f58-123bc69fdfc5_calibrate_1101466_50.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/1bbf4df7-aaf3-4d5b-8f58-123bc69fdfc5_calibrate_1101466_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:49 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbb74087e8505fe868a7c9f8c15fd9ae with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/0c92520d-031b-4078-bcf0-addf3fdcef00_calibrate_1101475_13.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/0c92520d-031b-4078-bcf0-addf3fdcef00_calibrate_1101475_13.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ecbb7eb6b01e43123a0526ff4752e1a with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2e72fed2-43c6-451d-b71a-a7d0590178a6_calibrate_1101466_3.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2e72fed2-43c6-451d-b71a-a7d0590178a6_calibrate_1101466_3.stderr parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0777655ae6517db6cba7e3f2c7423954 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/0e7af567-692e-4a25-8b21-226566fe46f6_calibrate_1101466_20.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/0e7af567-692e-4a25-8b21-226566fe46f6_calibrate_1101466_20.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f799ba378cf0fdf6395b3473c5d0475 with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2be8145d-c0eb-453f-99cd-72d6edbe70da_calibrate_1101466_46.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2be8145d-c0eb-453f-99cd-72d6edbe70da_calibrate_1101466_46.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b528db9a7b0f29c919c06b9426b4ba1 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/916ebda2-617e-4417-9041-e43e3e782b9c_calibrate_1101475_35.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/916ebda2-617e-4417-9041-e43e3e782b9c_calibrate_1101475_35.stderr parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b9c64b535c4263f78a8f06f6a72ae65 with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 3dfff9839d0c4c2a68b6743b200974a3 parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461609bf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/35ac050a-7f96-4ffe-ab4e-88ead7f2a652_calibrate_1101475_59.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/35ac050a-7f96-4ffe-ab4e-88ead7f2a652_calibrate_1101475_59.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a8bd12b7-946e-4015-84ca-202a827297d8_characterizeImage_1101475_59.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/a8bd12b7-946e-4015-84ca-202a827297d8_characterizeImage_1101475_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:50 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5262e615dbe417d06fe1ff8f128eeaf with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5fe4f49d-b466-4123-ae71-0c900fe94437_calibrate_1101475_54.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5fe4f49d-b466-4123-ae71-0c900fe94437_calibrate_1101475_54.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 972f5826e5addd0e37bbf1d51b2c4122 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/efcb7b7c-ecb7-4a74-9220-8357ae85d9cd_calibrate_1101475_50.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/efcb7b7c-ecb7-4a74-9220-8357ae85d9cd_calibrate_1101475_50.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa9c9909d5d6d705653c9313d8a80447 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/d971fe86-0040-4aea-9524-ba3bc3d02b58_calibrate_1101469_37.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/d971fe86-0040-4aea-9524-ba3bc3d02b58_calibrate_1101469_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:52 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7de96802104252cf4bfc9eb6a8dc89cf with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/eefeb1ca-1510-45bb-8b65-eba87f3bf573_calibrate_1101472_41.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/eefeb1ca-1510-45bb-8b65-eba87f3bf573_calibrate_1101472_41.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3fe1853091abbad82eba44d04d80572 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/a58145f2-045a-4f4b-bee6-b13261a59806_calibrate_1101469_43.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/a58145f2-045a-4f4b-bee6-b13261a59806_calibrate_1101469_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:54 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ea8d15001ce70ef6a92e4049a561e25 with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash b1596cbba90d5596e359bc2f1e2ad8ca parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b39440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d6d1906b-b98f-4d03-bcdf-4df990299b74_calibrate_1101475_21.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d6d1906b-b98f-4d03-bcdf-4df990299b74_calibrate_1101475_21.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e1d387f6-d655-4e30-86b2-4b813d8835ae_characterizeImage_1101475_21.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e1d387f6-d655-4e30-86b2-4b813d8835ae_characterizeImage_1101475_21.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dfa8ad5353694a06454675ecbf40b5d with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/2419fb2a-dfd4-41d2-ac70-ee5f346ba3de_calibrate_1101475_9.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/2419fb2a-dfd4-41d2-ac70-ee5f346ba3de_calibrate_1101475_9.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efaf01a3796f47cf0adeabc941f3ffc6 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/e17d54ea-bbf5-431f-923f-c3c9382b8687_calibrate_1101469_12.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/e17d54ea-bbf5-431f-923f-c3c9382b8687_calibrate_1101469_12.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b75c4abc1f95e605c61d5acb1871d9f4 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/58309d37-316b-406a-9c59-a1c6e5457ce7_calibrate_1101466_53.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/58309d37-316b-406a-9c59-a1c6e5457ce7_calibrate_1101466_53.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ec54367e7a5de71dcaaa5800f0d9102 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c3854330-45d4-4856-9177-945e01b6541e_calibrate_1101475_60.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c3854330-45d4-4856-9177-945e01b6541e_calibrate_1101475_60.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39bd489d88036ae682f4f663e3805552 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d3831056-42bc-436d-86cb-a18e31973495_calibrate_1101475_14.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d3831056-42bc-436d-86cb-a18e31973495_calibrate_1101475_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:56 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfdd88aea0cc947f0d6b173c877d58d9 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/478fbb0b-cc53-4dd6-8cc7-87bbbfa1d472_calibrate_1101466_44.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/478fbb0b-cc53-4dd6-8cc7-87bbbfa1d472_calibrate_1101466_44.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 000fa9271587734eb6a08da88736fd08 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8de0c1ea-a354-477c-aa7c-4a025ad94541_calibrate_1101469_9.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8de0c1ea-a354-477c-aa7c-4a025ad94541_calibrate_1101469_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:57 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85929ff12684b82b99b8c437edcca7a1 with result from task 260 parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:07:59 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b306772a-de12-4776-b94b-357351edae97_calibrate_1101466_37.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b306772a-de12-4776-b94b-357351edae97_calibrate_1101466_37.stderr parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52348789dd8f170c04b4a11308e25eb5 with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/c2f7e6bd-5b74-4a94-9b4e-d860ecd7ddb8_calibrate_1101469_31.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/c2f7e6bd-5b74-4a94-9b4e-d860ecd7ddb8_calibrate_1101469_31.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a7975e50d56d526bfce6d0bcb6ed3e2 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/f3990caa-7ece-4a3f-964f-81a6754c952d_calibrate_1101469_30.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/f3990caa-7ece-4a3f-964f-81a6754c952d_calibrate_1101469_30.stderr parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55e40ba7a2b8afef4a42cf55b85a040f with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4fff2529-1d44-437a-96e9-a921005cb5a2_calibrate_1101466_35.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4fff2529-1d44-437a-96e9-a921005cb5a2_calibrate_1101466_35.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2f51de0d3e9e4fd6310e03c318650c8 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/485fc6a3-2c53-493a-8a5b-d99838ac5d32_calibrate_1101472_6.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/485fc6a3-2c53-493a-8a5b-d99838ac5d32_calibrate_1101472_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:01 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8ae7a353fd7742d246bdccaf8cdef76 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/197c69b5-2919-42e6-bd7e-7943879e10f8_calibrate_1101472_1.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/197c69b5-2919-42e6-bd7e-7943879e10f8_calibrate_1101472_1.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0ced774d9af69a1375172e2ede28751 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/7d2e7469-7cd1-46ec-a5de-36785db09457_calibrate_1101475_58.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/7d2e7469-7cd1-46ec-a5de-36785db09457_calibrate_1101475_58.stderr parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84994da9792a187af7ad0df1d87718ce with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/cf1ca470-8238-4e5f-8318-e7b203cd7af4_calibrate_1101466_54.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/cf1ca470-8238-4e5f-8318-e7b203cd7af4_calibrate_1101466_54.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a0ee33fd955291354f65ff54f337770 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/a8d9c19c-5170-4c09-9615-055d4024c64e_calibrate_1101475_41.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/a8d9c19c-5170-4c09-9615-055d4024c64e_calibrate_1101475_41.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 721d3896c451e1fc6a9288cbd1b39bff with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c3f40ae4-763d-4e4e-976b-8a046de45702_calibrate_1101475_5.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c3f40ae4-763d-4e4e-976b-8a046de45702_calibrate_1101475_5.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a644360e78192c26721ae0766f05f5f1 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/d95ecf57-e62c-4d84-a19a-22cbb4fca3f7_calibrate_1101469_58.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/d95ecf57-e62c-4d84-a19a-22cbb4fca3f7_calibrate_1101469_58.stderr parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd1778f9a47f3c9c12b799434fd491a3 with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/07fdbf39-07c2-4865-bb1d-05ea985f165e_calibrate_1101466_28.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/07fdbf39-07c2-4865-bb1d-05ea985f165e_calibrate_1101466_28.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b92382bb93b60ad57eb964b718e908d5 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/48bbc288-7d95-46e0-ae49-dd50e63f2ad3_calibrate_1101469_18.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/48bbc288-7d95-46e0-ae49-dd50e63f2ad3_calibrate_1101469_18.stderr parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b689e645ec2d889a6eb2bef1a319ff39 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d2ffe84e-a8c1-4fbd-8a0e-f91f4f33ee31_calibrate_1101472_54.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d2ffe84e-a8c1-4fbd-8a0e-f91f4f33ee31_calibrate_1101472_54.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3793b19e305a98f465201dc3ee451e26 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/99845815-988c-4a0e-a04c-0aedf0f71d24_calibrate_1101469_62.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/99845815-988c-4a0e-a04c-0aedf0f71d24_calibrate_1101469_62.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c209cb61ae8a9a1a216e52203c96f28 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/05ef43c9-c195-4328-924e-a31d2394e9f0_calibrate_1101469_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/05ef43c9-c195-4328-924e-a31d2394e9f0_calibrate_1101469_41.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db243992e18064bfde3e53d50efed12a with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/79586ed5-1071-4013-ad6d-c0e37e7d088f_calibrate_1101475_44.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/79586ed5-1071-4013-ad6d-c0e37e7d088f_calibrate_1101475_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:05 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9403a387ea6ebfaa6a090dc2b4bd51b9 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/2b35b7d5-602d-4311-bbe3-a4d251a2ab2c_calibrate_1101475_29.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/2b35b7d5-602d-4311-bbe3-a4d251a2ab2c_calibrate_1101475_29.stderr parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e714a9e3d3f7a75f86473d7e9460a33 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2d46ea12-6516-4ed6-be76-7904ffeebffa_calibrate_1101466_24.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2d46ea12-6516-4ed6-be76-7904ffeebffa_calibrate_1101466_24.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43f6860c5d782d426258010f03ec38e7 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/9bbfd03f-f3d1-4d11-84ef-48ee2fbe0fab_calibrate_1101475_57.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/9bbfd03f-f3d1-4d11-84ef-48ee2fbe0fab_calibrate_1101475_57.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42a00b70494f4743b546ba6153492d45 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/17176af6-fc87-4c76-9539-9e25a523f718_calibrate_1101472_3.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/17176af6-fc87-4c76-9539-9e25a523f718_calibrate_1101472_3.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 463ba582be891a99eedc64a4ed3b2677 with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/270d0153-1458-4aa1-9432-4386b076adef_calibrate_1101466_56.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/270d0153-1458-4aa1-9432-4386b076adef_calibrate_1101466_56.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d38c7ff58ac488bf66f6a5442da0b99b with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/736b3a5d-61b9-4079-aa31-2696ea40850e_calibrate_1101472_43.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/736b3a5d-61b9-4079-aa31-2696ea40850e_calibrate_1101472_43.stderr parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc676f4e00d8b47f5984dcfe90ee00b4 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/19abd69c-9725-44d7-91ce-4ee55a2725c8_calibrate_1101469_45.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/19abd69c-9725-44d7-91ce-4ee55a2725c8_calibrate_1101469_45.stderr parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8609dbeb7ec25c6c888fb8c5e5432243 with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/bc2d84e2-1ea5-46e4-a74d-3a82af88b246_calibrate_1101469_44.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/bc2d84e2-1ea5-46e4-a74d-3a82af88b246_calibrate_1101469_44.stderr parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 415be7d084f19344618c9e42af617a86 with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ea4c88a5-ee17-4158-8b16-96a9e7f22bdc_calibrate_1101472_19.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ea4c88a5-ee17-4158-8b16-96a9e7f22bdc_calibrate_1101472_19.stderr parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7008ba897cf1ef7a164c151b7b7930c2 with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/217e953d-b0d9-4d98-9926-f5d53cfd172a_calibrate_1101466_40.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/217e953d-b0d9-4d98-9926-f5d53cfd172a_calibrate_1101466_40.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e934a2acf27eb64c1901a986124eae80 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/a0db8ced-8329-4109-915c-30f784677d54_calibrate_1101475_49.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/a0db8ced-8329-4109-915c-30f784677d54_calibrate_1101475_49.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3872ff72d6ea928881f481243c5b83e8 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/f4902252-cb6f-4fef-b2a1-cf421ed60551_calibrate_1101475_28.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/f4902252-cb6f-4fef-b2a1-cf421ed60551_calibrate_1101475_28.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb34e5691def00324b4137e18aeb30a5 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8809b9a9-df83-4e5c-9200-3b808821fa15_calibrate_1101469_46.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8809b9a9-df83-4e5c-9200-3b808821fa15_calibrate_1101469_46.stderr parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77eb61e261a7176d9d7f130239a131b0 with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/bc9c491e-2759-44be-a996-c329df0a7257_calibrate_1101466_55.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/bc9c491e-2759-44be-a996-c329df0a7257_calibrate_1101466_55.stderr parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b65097dd38f9a5bcbe078babc2b9383 with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/587f8cfb-3db2-472d-9471-c9257d75e64b_calibrate_1101469_23.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/587f8cfb-3db2-472d-9471-c9257d75e64b_calibrate_1101469_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:13 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 025eb22ee9de74bf472df6323ffd4589 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/608b444e-2983-478c-b0fe-e037192f4982_calibrate_1101466_41.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/608b444e-2983-478c-b0fe-e037192f4982_calibrate_1101466_41.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6c03e84427373321f2b36de88001192 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c7155b9e-b97b-4a03-8fca-8333ece6d872_calibrate_1101475_45.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c7155b9e-b97b-4a03-8fca-8333ece6d872_calibrate_1101475_45.stderr parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f8a1cc49116765051d31c2bd09b7064 with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/aa558952-0c75-43ab-9d40-3318703728a0_calibrate_1101466_13.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/aa558952-0c75-43ab-9d40-3318703728a0_calibrate_1101466_13.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 588521b41d848657eb3eaf04a2e5b1e6 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b5e1c444-60c0-47bd-9454-05cc387a112a_calibrate_1101466_25.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b5e1c444-60c0-47bd-9454-05cc387a112a_calibrate_1101466_25.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0db2c6cc9e7452e512ff83a7c286d0ed with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash e3e21e368df6739177b391c6a65e73cf parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b045e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/82b9fea4-f294-429c-9fc7-21025eeb5ce6_calibrate_1101466_51.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/82b9fea4-f294-429c-9fc7-21025eeb5ce6_calibrate_1101466_51.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/22edbfb2-39d7-40a8-8b7a-100370e41ed2_characterizeImage_1101466_51.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/22edbfb2-39d7-40a8-8b7a-100370e41ed2_characterizeImage_1101466_51.stderr parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 376c0a251c09bcb26c81c6f291658253 with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/11f5f2ec-9c78-4309-bb79-c9d5813291a0_calibrate_1101466_5.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/11f5f2ec-9c78-4309-bb79-c9d5813291a0_calibrate_1101466_5.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f50bf0a83f183ba3aee8ad194eff0f7 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/190833cf-05d4-4a80-ada2-59c7ad44ccfb_calibrate_1101472_13.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/190833cf-05d4-4a80-ada2-59c7ad44ccfb_calibrate_1101472_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:15 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97327e336993c77254724203fad139bf with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d533ac8a-86bf-436e-9d37-a68ba5108f7e_calibrate_1101472_33.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d533ac8a-86bf-436e-9d37-a68ba5108f7e_calibrate_1101472_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:17 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 830828e1d7b9570545d07f455569191a with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/24540e2f-9e59-4e46-9891-9dcc504c173e_calibrate_1101475_32.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/24540e2f-9e59-4e46-9891-9dcc504c173e_calibrate_1101475_32.stderr parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2faed1ca480ed964a1b7059f5f1c323 with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/8cd3c42f-3623-4d68-9bd5-db4985d7e7a5_calibrate_1101466_42.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/8cd3c42f-3623-4d68-9bd5-db4985d7e7a5_calibrate_1101466_42.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d2971bf82189a9c4ba600d1defc1767 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d9a32bd2-e021-4109-af68-0da6e43dbeca_calibrate_1101472_11.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d9a32bd2-e021-4109-af68-0da6e43dbeca_calibrate_1101472_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:18 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:20 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e3e6deb286724ce85b6313e209c5d43 with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/843dbc43-e951-43ae-9b50-79ab29a08d61_calibrate_1101472_12.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/843dbc43-e951-43ae-9b50-79ab29a08d61_calibrate_1101472_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:21 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 089f8d87fae18de8d3a19ec8186c6c28 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash dfb135cf6b6cf8669b339a1ebb86a878 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/bde30009-ffe2-4d16-b216-1257b10d976a_calibrate_1101472_61.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/bde30009-ffe2-4d16-b216-1257b10d976a_calibrate_1101472_61.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/0049625a-9cb3-41f5-8796-52f6d6126fb7_characterizeImage_1101472_61.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101472/0049625a-9cb3-41f5-8796-52f6d6126fb7_characterizeImage_1101472_61.stderr parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62bab524a5e1ed7a2f8829c7ce583c05 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/be30fd53-dded-431b-9f80-7e1ab572c6f8_calibrate_1101472_32.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/be30fd53-dded-431b-9f80-7e1ab572c6f8_calibrate_1101472_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:22 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 270bf72aeb2315f9d3d0e506e0b1e20e with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/8e709f50-2ec0-4cae-8328-7c1990e095f8_calibrate_1101475_16.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/8e709f50-2ec0-4cae-8328-7c1990e095f8_calibrate_1101475_16.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 465eaaed77928f8adf69474ebc69dd8a with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/22f2016b-d900-4f4b-81df-e2672b87a88a_calibrate_1101472_62.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/22f2016b-d900-4f4b-81df-e2672b87a88a_calibrate_1101472_62.stderr parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b038e83ff8b63c41d95539b582be287f with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash d3861f2a1d12a9933ab5b47bb6fffdaf parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a8e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d94c4664-1970-43ab-977b-a9f3641d1e85_calibrate_1101475_7.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d94c4664-1970-43ab-977b-a9f3641d1e85_calibrate_1101475_7.stderr parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e0e94eca-c0f5-49ce-983b-aece7e9b17a9_characterizeImage_1101475_7.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/e0e94eca-c0f5-49ce-983b-aece7e9b17a9_characterizeImage_1101475_7.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f3d16e04515ce2d5258ac26aa5944ed with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/90feaf23-93c7-49bb-8e4b-a408b1b96b15_calibrate_1101469_48.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/90feaf23-93c7-49bb-8e4b-a408b1b96b15_calibrate_1101469_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:24 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f57f29dc1c14f56e85a1e230813754b with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/e2a30825-1f2a-4ac0-a516-e269344f0824_calibrate_1101466_38.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/e2a30825-1f2a-4ac0-a516-e269344f0824_calibrate_1101466_38.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6116cfb4fd4261755e345a690fe55c0 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/c3c3ae2b-7519-4f8b-8044-5e477fa56f27_calibrate_1101472_42.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/c3c3ae2b-7519-4f8b-8044-5e477fa56f27_calibrate_1101472_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:26 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce48b37ccccd60fd102dc5c3c1dfc77e with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/6cd1a780-a96d-4915-a2e9-dc1d4bfcdff0_calibrate_1101475_47.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/6cd1a780-a96d-4915-a2e9-dc1d4bfcdff0_calibrate_1101475_47.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67073374039372c50d531337a03f5de3 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/7c5ebb93-35f5-4e66-a54b-e07f759a0a2d_calibrate_1101466_11.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/7c5ebb93-35f5-4e66-a54b-e07f759a0a2d_calibrate_1101466_11.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fccd2057f4dd02111c8824f71cffb29 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/126ecf71-70dd-4552-84b9-bc76346b98dc_calibrate_1101472_14.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/126ecf71-70dd-4552-84b9-bc76346b98dc_calibrate_1101472_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:27 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a7786e4799d6fbdcf835569d5c8f630 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/cfcd6480-a75f-40a1-b817-263a486b65ff_calibrate_1101475_18.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/cfcd6480-a75f-40a1-b817-263a486b65ff_calibrate_1101475_18.stderr parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62096bda1f83a0460bf255854b2b223f with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5cdfd107-c55b-41f6-b246-834dc0a56de6_calibrate_1101475_39.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5cdfd107-c55b-41f6-b246-834dc0a56de6_calibrate_1101475_39.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc4475f38d723b3f7d28a6e514ab0b98 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/792aa945-e8f5-4959-ab74-87e2dbadbc16_calibrate_1101472_4.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/792aa945-e8f5-4959-ab74-87e2dbadbc16_calibrate_1101472_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:29 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b614bc7c3941666432ed799af59b886 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/c81dc57e-5ecc-418a-91cc-854eaf29d85f_calibrate_1101469_25.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/c81dc57e-5ecc-418a-91cc-854eaf29d85f_calibrate_1101469_25.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8eb6543b042d222fcb31ff9df5a479a6 with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/f3ef0878-f074-4db2-9bfa-5feccddda07f_calibrate_1101475_38.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/f3ef0878-f074-4db2-9bfa-5feccddda07f_calibrate_1101475_38.stderr parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f48b0825ae177a40131f62512754547 with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/863d401d-16a4-46a8-afbe-78b20704607a_calibrate_1101466_8.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/863d401d-16a4-46a8-afbe-78b20704607a_calibrate_1101466_8.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84899033fbe631eb29908b8ea3cfbacb with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c6c4fb3f-38d3-45e2-859f-e852fad5ca21_calibrate_1101475_40.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c6c4fb3f-38d3-45e2-859f-e852fad5ca21_calibrate_1101475_40.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry deb56efd678818302b629acd1651ca10 with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d02fac2f-250c-413b-9a50-f48ea958eb5a_calibrate_1101472_57.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d02fac2f-250c-413b-9a50-f48ea958eb5a_calibrate_1101472_57.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec57f01c5548b6b5f4b9538744010da4 with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/01bdd519-625d-4be3-9126-63a4b0acfd80_calibrate_1101466_10.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/01bdd519-625d-4be3-9126-63a4b0acfd80_calibrate_1101466_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:31 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8469e505aa1a41be15f9894a8600531c with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4828693e-e189-4d8e-91d4-84cef0997bcc_calibrate_1101466_59.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4828693e-e189-4d8e-91d4-84cef0997bcc_calibrate_1101466_59.stderr parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9437b9ef7aa9a54a86a09b6023722ed3 with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2d408b9f-6fd6-4a76-b243-263c47a43ed0_calibrate_1101466_32.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2d408b9f-6fd6-4a76-b243-263c47a43ed0_calibrate_1101466_32.stderr parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1db55163ff6f787b016da39ae32d3869 with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/6f762e54-d73e-4dae-ae66-edd2a5c3ab44_calibrate_1101466_7.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/6f762e54-d73e-4dae-ae66-edd2a5c3ab44_calibrate_1101466_7.stderr parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b267e9b93227ec182a7142e9b41b1a46 with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/bc1b4c09-27ad-4413-8a8b-61b7d66aff56_calibrate_1101469_10.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/bc1b4c09-27ad-4413-8a8b-61b7d66aff56_calibrate_1101469_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:33 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d94c4924975c875fb7b6ba0aacc204f with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/9c374b7c-dd53-4b39-b199-0e1a91d6610a_calibrate_1101475_48.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/9c374b7c-dd53-4b39-b199-0e1a91d6610a_calibrate_1101475_48.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 632bc667be3817137e91a4742d063955 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/f09d1ab1-3d2b-4631-b6cc-b9d355c1bbba_calibrate_1101469_3.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/f09d1ab1-3d2b-4631-b6cc-b9d355c1bbba_calibrate_1101469_3.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4bfb0ec4c74a329352f2e200140b198 with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/b5ba0dea-c940-48b4-a7a7-95c68638c29a_calibrate_1101472_28.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/b5ba0dea-c940-48b4-a7a7-95c68638c29a_calibrate_1101472_28.stderr parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5119aa3666105019c8aac6c9c3f1d09 with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/588a91f0-75db-4ffc-b7b9-4a962db227cf_calibrate_1101466_15.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/588a91f0-75db-4ffc-b7b9-4a962db227cf_calibrate_1101466_15.stderr parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25f809a8f60a37e00d3dd781b1ffa7ef with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/9b9026a4-c866-4d0e-beb4-0cc7abc7f1da_calibrate_1101469_35.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/9b9026a4-c866-4d0e-beb4-0cc7abc7f1da_calibrate_1101469_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:36 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd7f79ccd81d9d4ea53f530aca3b46f with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/17cdd4a9-80c8-4ba7-830a-245b040380ea_calibrate_1101472_8.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/17cdd4a9-80c8-4ba7-830a-245b040380ea_calibrate_1101472_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:37 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b21eeca255a67693a8f924a7abd4db17 with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/da47c8e3-040f-4620-8d46-6dfc1f3d0fc0_calibrate_1101469_39.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/da47c8e3-040f-4620-8d46-6dfc1f3d0fc0_calibrate_1101469_39.stderr parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab2d4dd770a7a572c3386c6dfe6b2641 with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/f0cdf252-da50-4e3a-8639-7918d4719af3_calibrate_1101466_39.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/f0cdf252-da50-4e3a-8639-7918d4719af3_calibrate_1101466_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:40 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cd625dda284d88f0c88b81876ea3174 with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/8ab3f1d8-70f5-49b1-aa84-9ce17dc4b8b4_calibrate_1101466_23.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/8ab3f1d8-70f5-49b1-aa84-9ce17dc4b8b4_calibrate_1101466_23.stderr parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dde90ccce3e475383a4e9c01fe819745 with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/9d6e4339-020b-40f8-a1fc-80843588d9a3_calibrate_1101469_54.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/9d6e4339-020b-40f8-a1fc-80843588d9a3_calibrate_1101469_54.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2da41e965c20559a3ab9fc0f1d2462da with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/686bd3f0-b736-49c1-81aa-02c172060228_calibrate_1101472_27.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/686bd3f0-b736-49c1-81aa-02c172060228_calibrate_1101472_27.stderr parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cb431c8a20183c798cd64142f92fd9c with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/0e52825c-5be6-43f3-8909-81f30f3f7336_calibrate_1101472_20.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/0e52825c-5be6-43f3-8909-81f30f3f7336_calibrate_1101472_20.stderr parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ad44118e1d48e0ed7c7667a8b78c586 with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4500bbcf-6ca9-48c1-986b-3afda3084d99_calibrate_1101466_9.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4500bbcf-6ca9-48c1-986b-3afda3084d99_calibrate_1101466_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:41 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0cea21da5f90af48aab73ac120ac264 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/0e12f3b2-4aa3-43e6-827e-4d2cb05d86c9_calibrate_1101469_42.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/0e12f3b2-4aa3-43e6-827e-4d2cb05d86c9_calibrate_1101469_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:42 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 125f56a243f6568124b40788942df0b1 with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash 06beccf9c09a01ed1d01fca2e11f6b06 parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160a3c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/3dcefb2f-9692-47d5-81f8-4a1733f0ec6c_calibrate_1101466_36.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/3dcefb2f-9692-47d5-81f8-4a1733f0ec6c_calibrate_1101466_36.stderr parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/1fe46efa-688c-4c96-bec0-ae299db260a1_characterizeImage_1101466_36.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101466/1fe46efa-688c-4c96-bec0-ae299db260a1_characterizeImage_1101466_36.stderr parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2fdbd756d39879668e64d8f5468b35e with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/68f591f8-9859-4bb4-9253-05dc3e47b577_calibrate_1101466_22.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/68f591f8-9859-4bb4-9253-05dc3e47b577_calibrate_1101466_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:44 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5ad639b71e525cc73fadfbd3660c8df with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/43aff674-0e1e-442b-ac27-4ff219d26b3e_calibrate_1101475_19.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/43aff674-0e1e-442b-ac27-4ff219d26b3e_calibrate_1101475_19.stderr parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 102a1efa3b858047319fbe59711e4905 with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8a473e4e-206a-4140-a75c-17b5eb51c83c_calibrate_1101469_49.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8a473e4e-206a-4140-a75c-17b5eb51c83c_calibrate_1101469_49.stderr parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 757ae3907ceea09ea000cdea0c145745 with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d7f0a449-1fae-4599-af24-c5c19516340e_calibrate_1101472_50.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d7f0a449-1fae-4599-af24-c5c19516340e_calibrate_1101472_50.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6895b2efafea44bdf7ec0ce20d1f4437 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c382c125-ef19-4db1-9de7-c277b49e5c8e_calibrate_1101475_26.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c382c125-ef19-4db1-9de7-c277b49e5c8e_calibrate_1101475_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:45 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b19f5b7b5c65f5561cd4bf0ae531cfa with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash 19526940bcbc63fdab18ddf570a6d1d2 parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160b75260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/251472a8-3e23-47f7-9c25-27112c2cd3f9_calibrate_1101475_37.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/251472a8-3e23-47f7-9c25-27112c2cd3f9_calibrate_1101475_37.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/4ee82821-88e4-4398-adc4-a96d4b9dad39_characterizeImage_1101475_37.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/4ee82821-88e4-4398-adc4-a96d4b9dad39_characterizeImage_1101475_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:47 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64a89f61046da88bad7b5fd637626bb7 with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5cc2b083-0d73-47b4-8041-56db75780051_calibrate_1101466_58.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5cc2b083-0d73-47b4-8041-56db75780051_calibrate_1101466_58.stderr parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8b4ee4843ab2852ac0cfc75f2cf9aff with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/40ac9ba0-c6da-444a-8f3a-284455726bce_calibrate_1101466_43.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/40ac9ba0-c6da-444a-8f3a-284455726bce_calibrate_1101466_43.stderr parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5b9b241041ed1f31878ac6a87b0a2b6 with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/7799bbae-8aa1-4061-bc65-d6bd09721a50_calibrate_1101472_22.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/7799bbae-8aa1-4061-bc65-d6bd09721a50_calibrate_1101472_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:48 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c91c9f5da174742915b2c9f0a32bbda0 with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/c3e613c8-f4e4-49d9-a558-b730d087f1be_calibrate_1101472_55.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/c3e613c8-f4e4-49d9-a558-b730d087f1be_calibrate_1101472_55.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 011e3e3515930d392c0fced4f5cae13b with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c8ebcf9d-0acb-4e7b-a72c-23af62adf83b_calibrate_1101475_1.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c8ebcf9d-0acb-4e7b-a72c-23af62adf83b_calibrate_1101475_1.stderr parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6271864bb231b08a61118201409262f with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2a963dc7-1ca2-4ad5-9075-74f3c78e8de8_calibrate_1101466_47.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/2a963dc7-1ca2-4ad5-9075-74f3c78e8de8_calibrate_1101466_47.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdb0918be66b8f4a3b9caa184d04a407 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/9874250b-d333-4f8b-b3b6-80a190ce549c_calibrate_1101472_17.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/9874250b-d333-4f8b-b3b6-80a190ce549c_calibrate_1101472_17.stderr parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0679b19aa23620b5979672a11c81442e with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/9ac1aa47-cf5d-43dc-a46e-398673458c58_calibrate_1101475_8.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/9ac1aa47-cf5d-43dc-a46e-398673458c58_calibrate_1101475_8.stderr parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c957a818674d20d65b66945d9e65095a with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5df76fd5-7062-4fac-b4c1-0274831a2b20_calibrate_1101466_26.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/5df76fd5-7062-4fac-b4c1-0274831a2b20_calibrate_1101466_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:50 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a742506d55143c6b248982d381735c23 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/67a75334-339a-4466-916a-893485b53f23_calibrate_1101472_30.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/67a75334-339a-4466-916a-893485b53f23_calibrate_1101472_30.stderr parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08aec3ff51b10a0b46378c9818a2326c with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/b20586e2-aa0f-4d80-b797-9c8f95ee3ec2_calibrate_1101469_53.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/b20586e2-aa0f-4d80-b797-9c8f95ee3ec2_calibrate_1101469_53.stderr parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c35c660bb135ad292f3c27388400a2d with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2d35e606-6052-4db7-9aed-ffb04a8576ba_calibrate_1101469_24.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2d35e606-6052-4db7-9aed-ffb04a8576ba_calibrate_1101469_24.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 673b4477a784891864f57ec3e8da4807 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/14437f3a-b890-4597-8758-c163ee4a32f5_calibrate_1101475_22.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/14437f3a-b890-4597-8758-c163ee4a32f5_calibrate_1101475_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:52 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fb47b40ce8d0d8954e23938047d03e2 with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/1c9758cb-ea60-4b8b-b4e8-cea5701c78e8_calibrate_1101472_5.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/1c9758cb-ea60-4b8b-b4e8-cea5701c78e8_calibrate_1101472_5.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c480576e44668ddca5bbe9e4cc5cb8d with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/9ee46f26-ade8-42ac-b8e9-4ade7ad4f473_calibrate_1101472_23.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/9ee46f26-ade8-42ac-b8e9-4ade7ad4f473_calibrate_1101472_23.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 073d0c9bc55deb33dfaa134f2e4ffc48 with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/4cead598-7177-47ca-8d06-97e04668f9b4_calibrate_1101472_25.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/4cead598-7177-47ca-8d06-97e04668f9b4_calibrate_1101472_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:53 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90b3e10f7ec57d0783ce16e1b6a4bc67 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/24ec0977-5300-4de1-9f63-132425c433e2_calibrate_1101475_17.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/24ec0977-5300-4de1-9f63-132425c433e2_calibrate_1101475_17.stderr parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 027f8b4bdb4a8c059c0fb3431ffc401e with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7f6238ba-a64a-4904-bcbb-646d97f21dcc_calibrate_1101469_55.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7f6238ba-a64a-4904-bcbb-646d97f21dcc_calibrate_1101469_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:55 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4692d001bb1f99e86d9438bbd439ff0a with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/228eee85-f709-4e47-a918-a68898160c9c_calibrate_1101472_16.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/228eee85-f709-4e47-a918-a68898160c9c_calibrate_1101472_16.stderr parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80a5c25be3469cf891cb2eb6fd6e8e33 with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/1df71d75-fa80-4d12-97a6-47803efc1bef_calibrate_1101469_26.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/1df71d75-fa80-4d12-97a6-47803efc1bef_calibrate_1101469_26.stderr parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 476fa07b676eb13bb11b49cb331b805b with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4ad4e69a-b8de-427a-830e-98de5dbbe899_calibrate_1101466_62.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4ad4e69a-b8de-427a-830e-98de5dbbe899_calibrate_1101466_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:57 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdf654dcfe1afcac289c16f8718b9b6a with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/16c85dff-7257-4674-a02f-232b713af242_calibrate_1101466_21.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/16c85dff-7257-4674-a02f-232b713af242_calibrate_1101466_21.stderr parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 964165677bb768db7e827267562535db with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d47b7a3b-8d8c-4b86-97c0-147014599a64_calibrate_1101472_7.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d47b7a3b-8d8c-4b86-97c0-147014599a64_calibrate_1101472_7.stderr parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e294515a632ae4498fbf54fb691c64fb with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2f93d67a-369a-4dee-b32b-9cc8986514c1_calibrate_1101469_1.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2f93d67a-369a-4dee-b32b-9cc8986514c1_calibrate_1101469_1.stderr parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 416178ea005970cf8dc87cca41050a92 with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/90df92bc-a3ea-4c67-9d60-abd8c26e4b73_calibrate_1101472_35.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/90df92bc-a3ea-4c67-9d60-abd8c26e4b73_calibrate_1101472_35.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa540612811a4f2b354af992122ef1fd with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/4bbfc788-17a7-404b-8f46-adda138f1fbd_calibrate_1101469_4.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/4bbfc788-17a7-404b-8f46-adda138f1fbd_calibrate_1101469_4.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24bf51bdc0a23ce4487414bd6612036c with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/94f93ecf-3350-4dde-89c9-f641b48d17f8_calibrate_1101475_12.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/94f93ecf-3350-4dde-89c9-f641b48d17f8_calibrate_1101475_12.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7deda1e358722f86505e034674619eeb with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/3930ab43-b324-42b3-83b6-a3dc580d6033_calibrate_1101475_62.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/3930ab43-b324-42b3-83b6-a3dc580d6033_calibrate_1101475_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:08:59 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8368b439c0ccf41e98b00095ae2583ed with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/ded1ceb7-2dc8-464e-8a28-e439abc22e6c_calibrate_1101475_10.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/ded1ceb7-2dc8-464e-8a28-e439abc22e6c_calibrate_1101475_10.stderr parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99ad34e249d32538f48b6c9d3d5ebb57 with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/dccaf65e-741a-453f-b2ac-ba4fbade8aa6_calibrate_1101466_33.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/dccaf65e-741a-453f-b2ac-ba4fbade8aa6_calibrate_1101466_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:00 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98a4ccac1cceb7aaf862415e28dfe7cc with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/5ba8537f-7b5f-46df-85aa-462e690820d9_calibrate_1101472_47.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/5ba8537f-7b5f-46df-85aa-462e690820d9_calibrate_1101472_47.stderr parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 232ca84d35f36dd327ae9d29c4854392 with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ecea88b8-4629-47a1-a6bf-3b38e9f9b245_calibrate_1101472_26.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ecea88b8-4629-47a1-a6bf-3b38e9f9b245_calibrate_1101472_26.stderr parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b14523d6b351cd73e5eb8e59eaae3d2 with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/d9591c1e-2d16-4747-8021-c31adfb4c5b9_calibrate_1101466_4.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/d9591c1e-2d16-4747-8021-c31adfb4c5b9_calibrate_1101466_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:04 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dccd67df71106c60e7238de14f6883a6 with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/ac7f3fb2-10f9-45a3-9dc8-76dd51a6527c_calibrate_1101475_20.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/ac7f3fb2-10f9-45a3-9dc8-76dd51a6527c_calibrate_1101475_20.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a4202f85ee8f60fb0300c3b2211e971 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/332ccfd3-2323-4a4f-a7ad-5db8c2d879b3_calibrate_1101472_36.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/332ccfd3-2323-4a4f-a7ad-5db8c2d879b3_calibrate_1101472_36.stderr parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c580ca00c7a9da63b3998bdddab46bb1 with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b41a1686-b64d-4c96-9585-8ca3cb15917f_calibrate_1101466_45.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/b41a1686-b64d-4c96-9585-8ca3cb15917f_calibrate_1101466_45.stderr parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13a0f9e71af4a968c0983df856f7c968 with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/e6e89b66-001d-46bd-a0ab-ba944a180502_calibrate_1101469_52.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/e6e89b66-001d-46bd-a0ab-ba944a180502_calibrate_1101469_52.stderr parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f3ecfa387ff108eb039997730debb35 with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/33e3bd39-9466-45c8-b130-9f77c2eb23f0_calibrate_1101469_29.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/33e3bd39-9466-45c8-b130-9f77c2eb23f0_calibrate_1101469_29.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cf68ce7dd4f6a2f90245a5022f55951 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/41421b1e-fa98-4226-82a3-adba83b31982_calibrate_1101475_6.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/41421b1e-fa98-4226-82a3-adba83b31982_calibrate_1101475_6.stderr parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 792bdb65ad3da0d167e4c89833c6f7e4 with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/fab958d8-247c-4350-808a-7de4a4f48faa_calibrate_1101469_59.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/fab958d8-247c-4350-808a-7de4a4f48faa_calibrate_1101469_59.stderr parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 490ac97dd69030b0c40c3031cc499f3b with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ccdc191c-cc2c-4f46-8e7a-aa539d31dd04_calibrate_1101472_45.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ccdc191c-cc2c-4f46-8e7a-aa539d31dd04_calibrate_1101472_45.stderr parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7bcbbcc46c66b8f6bbcdb85ea5e91a8 with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4726763a-b245-41f0-9da1-3a509af4c58e_calibrate_1101466_48.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/4726763a-b245-41f0-9da1-3a509af4c58e_calibrate_1101466_48.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3e21e368df6739177b391c6a65e73cf with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/82b9fea4-f294-429c-9fc7-21025eeb5ce6_calibrate_1101466_51.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/82b9fea4-f294-429c-9fc7-21025eeb5ce6_calibrate_1101466_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:08 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5377e2be11a0cde1812e67cdcd58e35c with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/a10b4be4-75ac-4dd3-a718-a64ca8a10ced_calibrate_1101472_46.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/a10b4be4-75ac-4dd3-a718-a64ca8a10ced_calibrate_1101472_46.stderr parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c10e67afd2950584602f873eb0c78f7f with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8510e8eb-08cd-4199-8db4-cfc39c59c304_calibrate_1101469_28.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8510e8eb-08cd-4199-8db4-cfc39c59c304_calibrate_1101469_28.stderr parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e66df33a9f27a2d27e1f8360961eeab with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/0e2eaafc-f7ee-42da-ab1c-5a829b9eda05_calibrate_1101475_43.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/0e2eaafc-f7ee-42da-ab1c-5a829b9eda05_calibrate_1101475_43.stderr parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc013aa10e13bbd08cfbe7970259db69 with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/b7d09354-ade8-450e-bd88-832a1601b503_calibrate_1101469_6.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/b7d09354-ade8-450e-bd88-832a1601b503_calibrate_1101469_6.stderr parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a35189971549351656b5530d01dc940 with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/38656884-d96c-4535-a88d-150fb5b06457_calibrate_1101475_56.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/38656884-d96c-4535-a88d-150fb5b06457_calibrate_1101475_56.stderr parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 392621426fc9662a91fadac080be421d with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/108bdce6-133f-40b9-b1ff-723bcd72dc87_calibrate_1101472_10.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/108bdce6-133f-40b9-b1ff-723bcd72dc87_calibrate_1101472_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:10 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd2b5afb23048bd02209ba42892f36bb with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/73216ea5-b441-4e43-866d-204522a7587f_calibrate_1101475_15.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/73216ea5-b441-4e43-866d-204522a7587f_calibrate_1101475_15.stderr parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba32d62ca16a58aac9101aa4cd8f805b with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/13b1a916-fe38-4f2e-973b-856dba471436_calibrate_1101469_22.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/13b1a916-fe38-4f2e-973b-856dba471436_calibrate_1101469_22.stderr parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47829c94d159cf4b87eb314fe48eef68 with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/37be4d4c-0efd-474a-aa48-3a5985bd07af_calibrate_1101469_16.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/37be4d4c-0efd-474a-aa48-3a5985bd07af_calibrate_1101469_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:12 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 182 try 0 failed parsl.dataflow.dflow INFO: Task 182 marked for retry parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/bde30009-ffe2-4d16-b216-1257b10d976a_calibrate_1101472_61.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/bde30009-ffe2-4d16-b216-1257b10d976a_calibrate_1101472_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash dfb135cf6b6cf8669b339a1ebb86a878 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146160ab0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 1 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/bde30009-ffe2-4d16-b216-1257b10d976a_calibrate_1101472_61.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/bde30009-ffe2-4d16-b216-1257b10d976a_calibrate_1101472_61.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cd15cce77161f000931a0d9dced2d2d with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/652af4a9-8b60-4386-a72f-606540fe7df0_calibrate_1101472_9.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/652af4a9-8b60-4386-a72f-606540fe7df0_calibrate_1101472_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:13 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44688bec43125e0f34149ecb9288f896 with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/1b0701a4-e7cd-4ed4-8530-a5a826e83fe9_calibrate_1101475_23.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/1b0701a4-e7cd-4ed4-8530-a5a826e83fe9_calibrate_1101475_23.stderr parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 005c6993fe55dddbfa544ead6e03be9e with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8ef405d5-72f3-472d-9263-e0e5943bd894_calibrate_1101469_56.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/8ef405d5-72f3-472d-9263-e0e5943bd894_calibrate_1101469_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:15 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9b61258abcaf6dac028f8105a0852e0 with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/6f32275c-fec4-4c98-b7f9-72049872c6ae_calibrate_1101475_55.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/6f32275c-fec4-4c98-b7f9-72049872c6ae_calibrate_1101475_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:16 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02e9b22efdc1ea58fd0c999ecab600de with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/b14025d5-64b2-462c-aa38-0a3da52649c1_calibrate_1101472_48.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/b14025d5-64b2-462c-aa38-0a3da52649c1_calibrate_1101472_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:17 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2736030b796e8025e140424058c41443 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/383f8bbe-2aa2-47db-bf16-d0ff907555ae_calibrate_1101475_36.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/383f8bbe-2aa2-47db-bf16-d0ff907555ae_calibrate_1101475_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:18 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06beccf9c09a01ed1d01fca2e11f6b06 with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/3dcefb2f-9692-47d5-81f8-4a1733f0ec6c_calibrate_1101466_36.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/3dcefb2f-9692-47d5-81f8-4a1733f0ec6c_calibrate_1101466_36.stderr parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55a020d20d83c4158ac9baf5efbb2b46 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/1b6e3141-7003-4d1c-add6-236d8e2881d4_calibrate_1101472_38.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/1b6e3141-7003-4d1c-add6-236d8e2881d4_calibrate_1101472_38.stderr parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90eff0c1118bb4543899373cab9f67eb with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d2791c23-310d-4110-a0b8-ee462bf12901_calibrate_1101475_4.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d2791c23-310d-4110-a0b8-ee462bf12901_calibrate_1101475_4.stderr parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf6e7fcbd8446257f944d761e7a00183 with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/999118cc-5056-4195-8757-83936856b586_calibrate_1101475_34.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/999118cc-5056-4195-8757-83936856b586_calibrate_1101475_34.stderr parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0da14707bd6fbe77436bbb0596022651 with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7a9aef70-38b0-4c05-bcb1-8357804bd189_calibrate_1101469_36.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7a9aef70-38b0-4c05-bcb1-8357804bd189_calibrate_1101469_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:20 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 24 parsl.dataflow.strategy DEBUG: Executor multi has 24 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:09:21 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:09:21 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:23 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:23 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:25 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91228a1baef699aeb39676fd74986654 with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/bddeaad2-c731-4119-9fce-debc6c8a046a_calibrate_1101475_11.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/bddeaad2-c731-4119-9fce-debc6c8a046a_calibrate_1101475_11.stderr parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5b4f673a50cf16dc969bea133c7c5a2 with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/07a1240f-db2c-4cf6-bb3c-1cbc02c65f7d_calibrate_1101466_29.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101466/07a1240f-db2c-4cf6-bb3c-1cbc02c65f7d_calibrate_1101466_29.stderr parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61a6eaca5890806fba7cb20447f0fea9 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/38113d78-bba4-492d-bf5b-6cacb3e374b2_calibrate_1101475_3.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/38113d78-bba4-492d-bf5b-6cacb3e374b2_calibrate_1101475_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:25 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.memoization DEBUG: Storing app cache entry b6903257f78cf6012ef9e7878ee3e99a with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ba5434e9-5790-454f-aa4b-c67ea309a4d9_calibrate_1101472_44.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/ba5434e9-5790-454f-aa4b-c67ea309a4d9_calibrate_1101472_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:26 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc7b2d379c5e519dd5279744cacf289b with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/64bc9fe1-2992-4309-a53a-46e450ecde4a_calibrate_1101472_34.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/64bc9fe1-2992-4309-a53a-46e450ecde4a_calibrate_1101472_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:26 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:26 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 212156acde0a657425c38e6257b8b22d with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c7e309f4-2090-4919-b6e6-3d871312d099_calibrate_1101475_46.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/c7e309f4-2090-4919-b6e6-3d871312d099_calibrate_1101475_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:27 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0123948d78a9eafec3894073f4b6d21e with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d67c80b8-e0d6-4962-9305-74bc13d414fb_calibrate_1101472_53.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d67c80b8-e0d6-4962-9305-74bc13d414fb_calibrate_1101472_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:27 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:27 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:28 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e222e888de699e4df29871a5c01d9093 with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/02db57bc-3e56-4458-ba24-fa1232bd3f34_calibrate_1101475_42.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/02db57bc-3e56-4458-ba24-fa1232bd3f34_calibrate_1101475_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:29 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 893fae4dd497ae43997694199876e070 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 7b4aa1f8e0877115b6530f72a84caa83 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2d01c86b-6d1b-4034-92ca-5eb0c56118de_calibrate_1101469_60.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2d01c86b-6d1b-4034-92ca-5eb0c56118de_calibrate_1101469_60.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/dca6b1fc-d5fc-443c-9b14-ef43b7d121c1_characterizeImage_1101469_60.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/dca6b1fc-d5fc-443c-9b14-ef43b7d121c1_characterizeImage_1101469_60.stderr parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d38a22fa8aadc0c890971ef5c08c52c with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d625ea1f-bbb4-4e0a-881c-d7765eaac42b_calibrate_1101472_21.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/d625ea1f-bbb4-4e0a-881c-d7765eaac42b_calibrate_1101472_21.stderr parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03867228ac49e5e0eda4e010649fe943 with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5ff230f8-3735-447e-b9ce-51deb4ce4f99_calibrate_1101475_30.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/5ff230f8-3735-447e-b9ce-51deb4ce4f99_calibrate_1101475_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:31 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:32 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79cc997e5a021652df4783ae5f3eea29 with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/7c4eae54-2f04-43a9-bee7-d2cbd6019f2a_calibrate_1101472_37.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/7c4eae54-2f04-43a9-bee7-d2cbd6019f2a_calibrate_1101472_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:33 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd2ff24555c76af8bb2345a59bf92c55 with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/77bb4ae2-130c-4198-b092-c64604310dc3_calibrate_1101472_59.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/77bb4ae2-130c-4198-b092-c64604310dc3_calibrate_1101472_59.stderr parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dfff9839d0c4c2a68b6743b200974a3 with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/35ac050a-7f96-4ffe-ab4e-88ead7f2a652_calibrate_1101475_59.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/35ac050a-7f96-4ffe-ab4e-88ead7f2a652_calibrate_1101475_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:35 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ee62c2f8d847bd525e95f2d72c759f5 with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/f0d872a2-5f9c-4e08-ae2f-e400097f6aed_calibrate_1101469_21.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/f0d872a2-5f9c-4e08-ae2f-e400097f6aed_calibrate_1101469_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:36 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9781452c8765430f666c557de5005f04 with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/0ef2dbd1-956a-4f7b-a870-c27160dd0808_calibrate_1101475_33.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/0ef2dbd1-956a-4f7b-a870-c27160dd0808_calibrate_1101475_33.stderr parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e007fd96b8a05e6c594a57aadd243c14 with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/803ddc78-9b88-402b-9831-7ee3b8c454fe_calibrate_1101472_29.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/803ddc78-9b88-402b-9831-7ee3b8c454fe_calibrate_1101472_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:38 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:39 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb5315387290f6d54734e8a365d2b9fc with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/981cd3f4-bb85-4110-9604-2375ce760899_calibrate_1101472_60.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/981cd3f4-bb85-4110-9604-2375ce760899_calibrate_1101472_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:39 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:39 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:40 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:40 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:40 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:40 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:40 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:41 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:41 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:41 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:41 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:41 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:42 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:42 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:42 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:42 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:42 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:42 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:43 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:43 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:43 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:43 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:43 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1596cbba90d5596e359bc2f1e2ad8ca with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d6d1906b-b98f-4d03-bcdf-4df990299b74_calibrate_1101475_21.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d6d1906b-b98f-4d03-bcdf-4df990299b74_calibrate_1101475_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:44 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:44 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:44 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:44 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:45 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:45 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:46 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:46 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:46 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:46 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:46 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:46 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:46 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:47 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:47 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:47 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:47 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:47 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3861f2a1d12a9933ab5b47bb6fffdaf with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d94c4664-1970-43ab-977b-a9f3641d1e85_calibrate_1101475_7.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/d94c4664-1970-43ab-977b-a9f3641d1e85_calibrate_1101475_7.stderr 2024-03-19 01:09:52 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:09:52 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:53 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:53 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:53 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:53 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:53 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:53 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:54 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:54 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:54 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:54 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:54 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:54 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:55 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:55 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:55 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:55 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:55 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:55 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:56 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:56 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:56 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:56 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:56 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:56 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:57 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:57 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:57 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:57 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:57 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:57 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:57 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:58 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:58 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:09:59 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:01 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:02 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19526940bcbc63fdab18ddf570a6d1d2 with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/251472a8-3e23-47f7-9c25-27112c2cd3f9_calibrate_1101475_37.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/251472a8-3e23-47f7-9c25-27112c2cd3f9_calibrate_1101475_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:04 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:05 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:07 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:08 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:08 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:09 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:09 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:09 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:09 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:09 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:09 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:09 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:10 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:10 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:10 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:10 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:10 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:10 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:11 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:11 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:11 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 182 try 1 failed parsl.dataflow.dflow ERROR: Task 182 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry dfb135cf6b6cf8669b339a1ebb86a878 with result from task 182 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/bde30009-ffe2-4d16-b216-1257b10d976a_calibrate_1101472_61.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101472/bde30009-ffe2-4d16-b216-1257b10d976a_calibrate_1101472_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:11 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:11 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:11 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b4aa1f8e0877115b6530f72a84caa83 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2d01c86b-6d1b-4034-92ca-5eb0c56118de_calibrate_1101469_60.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/2d01c86b-6d1b-4034-92ca-5eb0c56118de_calibrate_1101469_60.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16a74798baee91ff29d5b263b0011518 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 894947168232bb5e94270e4280af64bc parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1461620bef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/243ca9df-bfe9-477f-911c-4ac86d9a078d_calibrate_1101475_61.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/243ca9df-bfe9-477f-911c-4ac86d9a078d_calibrate_1101475_61.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/d4add97f-5875-46d5-a9ff-9440e608356c_characterizeImage_1101475_61.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101475/d4add97f-5875-46d5-a9ff-9440e608356c_characterizeImage_1101475_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:10:26 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:10:27 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:27 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:27 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:27 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:27 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:28 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:28 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:28 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:28 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:28 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:28 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:29 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:29 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:29 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:29 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:29 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:29 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:30 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:30 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:30 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:30 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:30 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:30 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:31 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:31 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:31 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:31 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:31 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:31 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:32 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:32 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:32 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:32 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:32 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:32 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:33 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:33 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:33 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:33 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:33 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:34 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:34 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:34 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:34 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:34 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:34 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:35 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:35 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:35 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:35 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:35 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:35 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:36 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:36 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:36 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:36 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:36 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:36 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:37 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:37 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:37 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:10:57 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:10:57 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:57 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:57 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:58 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:58 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:58 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:58 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:58 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:58 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:59 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:59 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:59 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:59 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:59 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:59 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:10:59 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:00 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:00 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:00 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:00 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:00 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:00 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:01 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:01 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:01 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:01 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:01 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:01 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:01 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:02 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:02 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:02 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:02 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:02 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:02 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:02 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:03 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:03 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:03 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:03 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:03 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:03 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:04 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:04 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 894947168232bb5e94270e4280af64bc with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/243ca9df-bfe9-477f-911c-4ac86d9a078d_calibrate_1101475_61.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101475/243ca9df-bfe9-477f-911c-4ac86d9a078d_calibrate_1101475_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:04 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:04 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:04 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:04 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:05 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:05 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:05 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:05 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:05 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:05 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:05 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:06 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:06 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:06 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:06 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:06 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:06 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:11:32 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:11:32 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:32 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:32 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:32 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:32 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:32 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:33 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:33 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:33 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:33 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:33 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:33 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:34 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:34 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:34 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:34 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:34 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:34 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:35 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:35 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:35 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:35 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:35 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:35 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:36 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:36 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:36 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:36 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:36 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:36 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:37 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:37 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:37 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:37 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:37 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:37 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:38 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:38 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:38 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:38 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:38 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:38 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:39 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:39 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:39 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:39 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:39 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:39 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:40 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:40 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:40 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:40 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:40 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:40 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:41 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:41 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:41 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:41 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:41 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:41 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:11:42 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:12:02 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:12:02 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:02 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:02 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:02 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:03 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:03 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:03 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:03 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:03 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:03 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:04 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:04 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:04 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:04 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:04 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:04 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:04 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:05 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:05 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:05 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:05 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:05 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:05 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:06 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:06 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:06 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:06 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:06 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:06 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:07 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:07 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:07 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:07 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:07 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:07 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:08 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:08 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:08 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:08 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:08 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:08 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:09 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:09 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:09 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:09 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:09 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:09 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:10 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:10 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:10 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:10 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:10 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:10 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:11 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:11 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:11 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:11 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:11 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:11 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:12 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:12 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:12:32 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:12:32 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:32 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:32 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:32 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:33 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:33 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:33 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:33 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:33 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:33 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:34 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:34 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:34 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:34 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:34 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:34 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:35 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:35 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:35 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:35 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:35 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:35 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:36 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:36 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:36 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:36 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:36 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:36 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:37 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:37 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:37 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:37 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:37 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:37 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:37 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:38 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:38 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:38 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:38 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:38 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:38 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:39 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:39 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:39 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:40 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:40 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:40 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:40 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:40 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:40 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:41 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:41 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:41 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:41 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:41 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:42 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:42 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:42 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:42 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:42 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:12:42 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:13:03 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:13:03 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:03 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:03 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:03 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:03 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:03 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:04 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:04 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:04 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:04 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:04 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:04 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:05 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:05 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:05 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:05 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:05 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:05 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:06 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:06 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:06 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:06 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:06 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:06 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:07 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:07 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:07 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:07 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:07 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:07 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:07 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:08 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:08 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:08 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:08 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:08 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:08 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:09 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:09 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:09 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:09 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:09 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:09 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:10 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:10 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:10 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:10 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:10 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:11 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:11 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:11 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:11 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:11 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:11 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:12 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:12 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:12 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:12 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:12 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:12 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:13 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a339c2703ba7e4a66e3291cb3a151be with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 21522b37d8eaa1388a4f4044f3484881 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7d865684-e83c-4128-acf8-411497b7d293_calibrate_1101469_61.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7d865684-e83c-4128-acf8-411497b7d293_calibrate_1101469_61.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/969bf8b1-bf98-41b8-83d3-deb012d8f271_characterizeImage_1101469_61.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/characterizeImage/1101469/969bf8b1-bf98-41b8-83d3-deb012d8f271_characterizeImage_1101469_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 2024-03-19 01:13:33 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:13:33 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:33 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:33 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:33 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:34 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:34 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:34 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:34 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:34 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:34 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:35 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:35 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:35 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:35 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:35 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:36 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:36 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:36 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:36 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:36 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:37 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:37 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:37 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:37 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:38 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:38 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:38 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:38 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:38 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:39 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226175,17226176,17226177,17226178,17226179,17226180,17226181,17226182,17226183,17226184,17226185,17226186,17226187,17226188,17226189,17226190,17226191,17226192,17226193,17226194,17226195,17226196,17226197,17226198,17226199,17226200,17226201,17226202,17226203,17226204' parsl.providers.slurm.slurm DEBUG: squeue returned 17226197 R 17226202 R 17226203 R 17226204 R 17226187 R 17226188 R 17226189 R 17226190 R 17226191 R 17226192 R 17226193 R 17226194 R 17226195 R 17226196 R 17226198 R 17226199 R 17226200 R 17226201 R 17226177 R 17226178 R 17226179 R 17226180 R 17226181 R 17226182 R 17226183 R 17226184 R 17226185 R 17226186 R 17226175 R 17226176 R parsl.providers.slurm.slurm DEBUG: Updating job 17226197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226198 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226176 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:39 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:39 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:39 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:39 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:40 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226205,17226206,17226207,17226208,17226209' parsl.providers.slurm.slurm DEBUG: squeue returned 17226208 R 17226209 R 17226205 R 17226206 R 17226207 R parsl.providers.slurm.slurm DEBUG: Updating job 17226208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226207 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:40 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:40 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:40 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:41 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:41 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:41 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:41 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 80 try 0 failed parsl.dataflow.dflow INFO: Task 80 marked for retry parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7d865684-e83c-4128-acf8-411497b7d293_calibrate_1101469_61.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7d865684-e83c-4128-acf8-411497b7d293_calibrate_1101469_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 21522b37d8eaa1388a4f4044f3484881 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14616209a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 1 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7d865684-e83c-4128-acf8-411497b7d293_calibrate_1101469_61.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7d865684-e83c-4128-acf8-411497b7d293_calibrate_1101469_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:41 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:42 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:42 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:42 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:42 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:42 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:42 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:43 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:43 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:43 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:43 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:43 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:43 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226210,17226211,17226212,17226213,17226214,17226215,17226216,17226217,17226218,17226219,17226220,17226221,17226222,17226223,17226224,17226225,17226226,17226227,17226228,17226229' parsl.providers.slurm.slurm DEBUG: squeue returned 17226227 R 17226228 R 17226229 R 17226224 R 17226225 R 17226226 R 17226221 R 17226222 R 17226223 R 17226218 R 17226219 R 17226220 R 17226213 R 17226214 R 17226215 R 17226216 R 17226217 R 17226210 R 17226211 R 17226212 R parsl.providers.slurm.slurm DEBUG: Updating job 17226227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226212 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:44 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:44 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:44 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:44 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:44 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING 2024-03-19 01:13:44 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226230,17226231,17226232,17226233,17226237,17226238' parsl.providers.slurm.slurm DEBUG: squeue returned 17226233 R 17226237 R 17226238 R 17226230 R 17226231 R 17226232 R parsl.providers.slurm.slurm DEBUG: Updating job 17226233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226232 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22408504651088 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 80 try 1 failed parsl.dataflow.dflow ERROR: Task 80 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 21522b37d8eaa1388a4f4044f3484881 with result from task 80 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 732 parsl.dataflow.dflow INFO: Tasks in state States.failed: 7 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 5 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7d865684-e83c-4128-acf8-411497b7d293_calibrate_1101469_61.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/submit/DEEP/20220527/A0c/science#step1/20240319T010033Z/logs/calibrate/1101469/7d865684-e83c-4128-acf8-411497b7d293_calibrate_1101469_61.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=62, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 62 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3ede28a1331b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3ede28a1331b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ae0dc89cdf32 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ae0dc89cdf32 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b40a23256eff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b40a23256eff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 31299ac7eb07 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 31299ac7eb07 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e2ad6d00c319 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e2ad6d00c319 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5f2ccebd5237 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5f2ccebd5237 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3b9904319fac parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3b9904319fac parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8f56b7106a90 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8f56b7106a90 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 508067b835f9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 508067b835f9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c284054bd8de parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c284054bd8de parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 22e9453d8c07 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 22e9453d8c07 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 33691ec0049f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 33691ec0049f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c32d0e3bec89 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c32d0e3bec89 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3c3551faf88a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3c3551faf88a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 70ccf691c11d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 70ccf691c11d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1101e27d6856 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1101e27d6856 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e4eeeda2085e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e4eeeda2085e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 40a06af61358 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 40a06af61358 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6ac4581ee781 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6ac4581ee781 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9e8c1c4653a9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9e8c1c4653a9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e3d7eaef8d67 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e3d7eaef8d67 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7f830f45042e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7f830f45042e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cd312b19130e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cd312b19130e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 17ef559335ec parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 17ef559335ec parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4906ebd796db parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4906ebd796db parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 919e57f42aef parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 919e57f42aef parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 27e06d26a5b0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 27e06d26a5b0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e28a2787155f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e28a2787155f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9cc2fe2abad7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9cc2fe2abad7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2cecc77ffb9d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2cecc77ffb9d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7788f41657ff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7788f41657ff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c7de9f46574e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c7de9f46574e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 066d07013f93 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 066d07013f93 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 52c9f15a44b8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 52c9f15a44b8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cee31ee7b44a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cee31ee7b44a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dd4dd2d85d2a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dd4dd2d85d2a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1fe0301fe370 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1fe0301fe370 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 27789d920b43 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 27789d920b43 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 01973195ff63 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 01973195ff63 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4daff39840be parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4daff39840be parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b460ac1152b2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b460ac1152b2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0ab55f9141da parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0ab55f9141da parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5fe096125500 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5fe096125500 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a92c89d14cba parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a92c89d14cba parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bb19bcd61732 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bb19bcd61732 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d8ece698e63 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d8ece698e63 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 56402b0fcc10 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 56402b0fcc10 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3de6e60efa2d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3de6e60efa2d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d3a9c23b3d38 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d3a9c23b3d38 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c2a2e2bb9d6e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c2a2e2bb9d6e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 49256321e04f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 49256321e04f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d6d4ce0dca96 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d6d4ce0dca96 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4a55e31a60ee parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4a55e31a60ee parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 441ac2e80acd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 441ac2e80acd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 797092cbc58b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 797092cbc58b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c13d5bec2bde parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c13d5bec2bde parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 94dda9dc9bff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 94dda9dc9bff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f17340edfbfd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f17340edfbfd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 77ed2c6a997c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 77ed2c6a997c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d92ae718747 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d92ae718747 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f0c010727c6a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f0c010727c6a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1ae456523341 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1ae456523341 2024-03-19 01:13:58 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:13:58 proc_lsst.multi:201 [INFO] cancelling 25873 on provider local proc_lsst.multi INFO: cancelling 25873 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 25873 2024-03-19 01:13:59 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro 2024-03-19 01:13:59 proc_lsst.multi:201 [INFO] cancelling 17226190 on provider astro proc_lsst.multi INFO: cancelling 17226190 on provider astro 2024-03-19 01:13:59 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro 2024-03-19 01:13:59 proc_lsst.multi:201 [INFO] cancelling 17226177 on provider astro proc_lsst.multi INFO: cancelling 17226177 on provider astro 2024-03-19 01:13:59 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint 2024-03-19 01:13:59 proc_lsst.multi:201 [INFO] cancelling 17226230 on provider checkpoint proc_lsst.multi INFO: cancelling 17226230 on provider checkpoint 2024-03-19 01:13:59 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro 2024-03-19 01:13:59 proc_lsst.multi:201 [INFO] cancelling 17226179 on provider astro proc_lsst.multi INFO: cancelling 17226179 on provider astro 2024-03-19 01:13:59 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 2024-03-19 01:13:59 proc_lsst.multi:201 [INFO] cancelling 17226213 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226213 on provider escience_a40 2024-03-19 01:13:59 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro 2024-03-19 01:13:59 proc_lsst.multi:201 [INFO] cancelling 17226180 on provider astro proc_lsst.multi INFO: cancelling 17226180 on provider astro 2024-03-19 01:13:59 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro 2024-03-19 01:13:59 proc_lsst.multi:201 [INFO] cancelling 17226191 on provider astro proc_lsst.multi INFO: cancelling 17226191 on provider astro 2024-03-19 01:13:59 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro 2024-03-19 01:13:59 proc_lsst.multi:201 [INFO] cancelling 17226181 on provider astro proc_lsst.multi INFO: cancelling 17226181 on provider astro 2024-03-19 01:13:59 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro 2024-03-19 01:13:59 proc_lsst.multi:201 [INFO] cancelling 17226196 on provider astro proc_lsst.multi INFO: cancelling 17226196 on provider astro 2024-03-19 01:14:00 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k 2024-03-19 01:14:00 proc_lsst.multi:201 [INFO] cancelling 17226206 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17226206 on provider escience_rtx6k 2024-03-19 01:14:00 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint 2024-03-19 01:14:00 proc_lsst.multi:201 [INFO] cancelling 17226231 on provider checkpoint proc_lsst.multi INFO: cancelling 17226231 on provider checkpoint 2024-03-19 01:14:00 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint 2024-03-19 01:14:00 proc_lsst.multi:201 [INFO] cancelling 17226233 on provider checkpoint proc_lsst.multi INFO: cancelling 17226233 on provider checkpoint 2024-03-19 01:14:00 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 2024-03-19 01:14:00 proc_lsst.multi:201 [INFO] cancelling 17226223 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226223 on provider escience_a40 2024-03-19 01:14:00 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro 2024-03-19 01:14:00 proc_lsst.multi:201 [INFO] cancelling 17226204 on provider astro proc_lsst.multi INFO: cancelling 17226204 on provider astro 2024-03-19 01:14:00 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro 2024-03-19 01:14:00 proc_lsst.multi:201 [INFO] cancelling 17226199 on provider astro proc_lsst.multi INFO: cancelling 17226199 on provider astro 2024-03-19 01:14:00 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint 2024-03-19 01:14:00 proc_lsst.multi:201 [INFO] cancelling 17226232 on provider checkpoint proc_lsst.multi INFO: cancelling 17226232 on provider checkpoint 2024-03-19 01:14:00 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k 2024-03-19 01:14:00 proc_lsst.multi:201 [INFO] cancelling 17226208 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17226208 on provider escience_rtx6k 2024-03-19 01:14:00 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 2024-03-19 01:14:00 proc_lsst.multi:201 [INFO] cancelling 17226221 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226221 on provider escience_a40 2024-03-19 01:14:01 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro 2024-03-19 01:14:01 proc_lsst.multi:201 [INFO] cancelling 17226182 on provider astro proc_lsst.multi INFO: cancelling 17226182 on provider astro 2024-03-19 01:14:01 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro 2024-03-19 01:14:01 proc_lsst.multi:201 [INFO] cancelling 17226203 on provider astro proc_lsst.multi INFO: cancelling 17226203 on provider astro 2024-03-19 01:14:01 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k 2024-03-19 01:14:01 proc_lsst.multi:201 [INFO] cancelling 17226209 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17226209 on provider escience_rtx6k 2024-03-19 01:14:01 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint 2024-03-19 01:14:01 proc_lsst.multi:201 [INFO] cancelling 17226237 on provider checkpoint proc_lsst.multi INFO: cancelling 17226237 on provider checkpoint 2024-03-19 01:14:01 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 2024-03-19 01:14:01 proc_lsst.multi:201 [INFO] cancelling 17226218 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226218 on provider escience_a40 2024-03-19 01:14:01 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint 2024-03-19 01:14:01 proc_lsst.multi:201 [INFO] cancelling 17226238 on provider checkpoint proc_lsst.multi INFO: cancelling 17226238 on provider checkpoint 2024-03-19 01:14:01 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro 2024-03-19 01:14:01 proc_lsst.multi:201 [INFO] cancelling 17226195 on provider astro proc_lsst.multi INFO: cancelling 17226195 on provider astro 2024-03-19 01:14:01 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 2024-03-19 01:14:01 proc_lsst.multi:201 [INFO] cancelling 17226219 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226219 on provider escience_a40 2024-03-19 01:14:01 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k 2024-03-19 01:14:01 proc_lsst.multi:201 [INFO] cancelling 17226205 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17226205 on provider escience_rtx6k 2024-03-19 01:14:02 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro 2024-03-19 01:14:02 proc_lsst.multi:201 [INFO] cancelling 17226198 on provider astro proc_lsst.multi INFO: cancelling 17226198 on provider astro 2024-03-19 01:14:02 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 2024-03-19 01:14:02 proc_lsst.multi:201 [INFO] cancelling 17226216 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226216 on provider escience_a40 2024-03-19 01:14:02 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro 2024-03-19 01:14:02 proc_lsst.multi:201 [INFO] cancelling 17226175 on provider astro proc_lsst.multi INFO: cancelling 17226175 on provider astro 2024-03-19 01:14:02 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 2024-03-19 01:14:02 proc_lsst.multi:201 [INFO] cancelling 17226215 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226215 on provider escience_a40 2024-03-19 01:14:02 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 2024-03-19 01:14:02 proc_lsst.multi:201 [INFO] cancelling 17226225 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226225 on provider escience_a40 2024-03-19 01:14:02 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro 2024-03-19 01:14:02 proc_lsst.multi:201 [INFO] cancelling 17226188 on provider astro proc_lsst.multi INFO: cancelling 17226188 on provider astro 2024-03-19 01:14:02 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 2024-03-19 01:14:02 proc_lsst.multi:201 [INFO] cancelling 17226217 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226217 on provider escience_a40 2024-03-19 01:14:02 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro 2024-03-19 01:14:02 proc_lsst.multi:201 [INFO] cancelling 17226197 on provider astro proc_lsst.multi INFO: cancelling 17226197 on provider astro 2024-03-19 01:14:02 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro 2024-03-19 01:14:02 proc_lsst.multi:201 [INFO] cancelling 17226178 on provider astro proc_lsst.multi INFO: cancelling 17226178 on provider astro 2024-03-19 01:14:03 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 2024-03-19 01:14:03 proc_lsst.multi:201 [INFO] cancelling 17226214 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226214 on provider escience_a40 2024-03-19 01:14:03 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 2024-03-19 01:14:03 proc_lsst.multi:201 [INFO] cancelling 17226228 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226228 on provider escience_a40 2024-03-19 01:14:03 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro 2024-03-19 01:14:03 proc_lsst.multi:201 [INFO] cancelling 17226189 on provider astro proc_lsst.multi INFO: cancelling 17226189 on provider astro 2024-03-19 01:14:03 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro 2024-03-19 01:14:03 proc_lsst.multi:201 [INFO] cancelling 17226193 on provider astro proc_lsst.multi INFO: cancelling 17226193 on provider astro 2024-03-19 01:14:03 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 2024-03-19 01:14:03 proc_lsst.multi:201 [INFO] cancelling 17226229 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226229 on provider escience_a40 2024-03-19 01:14:03 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro 2024-03-19 01:14:03 proc_lsst.multi:201 [INFO] cancelling 17226184 on provider astro proc_lsst.multi INFO: cancelling 17226184 on provider astro 2024-03-19 01:14:03 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 2024-03-19 01:14:03 proc_lsst.multi:201 [INFO] cancelling 17226227 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226227 on provider escience_a40 2024-03-19 01:14:04 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro 2024-03-19 01:14:04 proc_lsst.multi:201 [INFO] cancelling 17226185 on provider astro proc_lsst.multi INFO: cancelling 17226185 on provider astro 2024-03-19 01:14:04 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro 2024-03-19 01:14:04 proc_lsst.multi:201 [INFO] cancelling 17226194 on provider astro proc_lsst.multi INFO: cancelling 17226194 on provider astro 2024-03-19 01:14:04 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 2024-03-19 01:14:04 proc_lsst.multi:201 [INFO] cancelling 17226222 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226222 on provider escience_a40 2024-03-19 01:14:04 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro 2024-03-19 01:14:04 proc_lsst.multi:201 [INFO] cancelling 17226183 on provider astro proc_lsst.multi INFO: cancelling 17226183 on provider astro 2024-03-19 01:14:04 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro 2024-03-19 01:14:04 proc_lsst.multi:201 [INFO] cancelling 17226186 on provider astro proc_lsst.multi INFO: cancelling 17226186 on provider astro 2024-03-19 01:14:04 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 2024-03-19 01:14:04 proc_lsst.multi:201 [INFO] cancelling 17226212 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226212 on provider escience_a40 2024-03-19 01:14:04 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 2024-03-19 01:14:04 proc_lsst.multi:201 [INFO] cancelling 17226220 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226220 on provider escience_a40 2024-03-19 01:14:05 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k 2024-03-19 01:14:05 proc_lsst.multi:201 [INFO] cancelling 17226207 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17226207 on provider escience_rtx6k 2024-03-19 01:14:05 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 2024-03-19 01:14:05 proc_lsst.multi:201 [INFO] cancelling 17226226 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226226 on provider escience_a40 2024-03-19 01:14:05 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 2024-03-19 01:14:05 proc_lsst.multi:201 [INFO] cancelling 17226210 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226210 on provider escience_a40 2024-03-19 01:14:05 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro 2024-03-19 01:14:05 proc_lsst.multi:201 [INFO] cancelling 17226202 on provider astro proc_lsst.multi INFO: cancelling 17226202 on provider astro 2024-03-19 01:14:05 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 2024-03-19 01:14:05 proc_lsst.multi:201 [INFO] cancelling 17226224 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226224 on provider escience_a40 2024-03-19 01:14:05 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro 2024-03-19 01:14:05 proc_lsst.multi:201 [INFO] cancelling 17226176 on provider astro proc_lsst.multi INFO: cancelling 17226176 on provider astro 2024-03-19 01:14:05 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro 2024-03-19 01:14:05 proc_lsst.multi:201 [INFO] cancelling 17226201 on provider astro proc_lsst.multi INFO: cancelling 17226201 on provider astro 2024-03-19 01:14:06 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 2024-03-19 01:14:06 proc_lsst.multi:201 [INFO] cancelling 17226211 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226211 on provider escience_a40 2024-03-19 01:14:06 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro 2024-03-19 01:14:06 proc_lsst.multi:201 [INFO] cancelling 17226200 on provider astro proc_lsst.multi INFO: cancelling 17226200 on provider astro 2024-03-19 01:14:06 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro 2024-03-19 01:14:06 proc_lsst.multi:201 [INFO] cancelling 17226187 on provider astro proc_lsst.multi INFO: cancelling 17226187 on provider astro 2024-03-19 01:14:06 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro 2024-03-19 01:14:06 proc_lsst.multi:201 [INFO] cancelling 17226192 on provider astro proc_lsst.multi INFO: cancelling 17226192 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-19 01:14:06 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-19 01:14:06 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17226230', '17226231', '17226232', '17226233', '17226237', '17226238', '17226175', '17226176', '17226177', '17226178', '17226179', '17226180', '17226181', '17226182', '17226183', '17226184', '17226185', '17226186', '17226187', '17226188', '17226189', '17226190', '17226191', '17226192', '17226193', '17226194', '17226195', '17226196', '17226197', '17226198', '17226199', '17226200', '17226201', '17226202', '17226203', '17226204', '17226210', '17226211', '17226212', '17226213', '17226214', '17226215', '17226216', '17226217', '17226218', '17226219', '17226220', '17226221', '17226222', '17226223', '17226224', '17226225', '17226226', '17226227', '17226228', '17226229', '17226205', '17226206', '17226207', '17226208', '17226209', '25873'] proc_lsst.multi INFO: new jobs since last cancel ['17226230', '17226231', '17226232', '17226233', '17226237', '17226238', '17226175', '17226176', '17226177', '17226178', '17226179', '17226180', '17226181', '17226182', '17226183', '17226184', '17226185', '17226186', '17226187', '17226188', '17226189', '17226190', '17226191', '17226192', '17226193', '17226194', '17226195', '17226196', '17226197', '17226198', '17226199', '17226200', '17226201', '17226202', '17226203', '17226204', '17226210', '17226211', '17226212', '17226213', '17226214', '17226215', '17226216', '17226217', '17226218', '17226219', '17226220', '17226221', '17226222', '17226223', '17226224', '17226225', '17226226', '17226227', '17226228', '17226229', '17226205', '17226206', '17226207', '17226208', '17226209', '25873'] 2024-03-19 01:14:06 proc_lsst.multi:146 [INFO] found job 17226230 in provider checkpoint proc_lsst.multi INFO: found job 17226230 in provider checkpoint 2024-03-19 01:14:06 proc_lsst.multi:201 [INFO] cancelling 17226230 on provider checkpoint proc_lsst.multi INFO: cancelling 17226230 on provider checkpoint 2024-03-19 01:14:06 proc_lsst.multi:146 [INFO] found job 17226231 in provider checkpoint proc_lsst.multi INFO: found job 17226231 in provider checkpoint 2024-03-19 01:14:06 proc_lsst.multi:201 [INFO] cancelling 17226231 on provider checkpoint proc_lsst.multi INFO: cancelling 17226231 on provider checkpoint 2024-03-19 01:14:06 proc_lsst.multi:146 [INFO] found job 17226232 in provider checkpoint proc_lsst.multi INFO: found job 17226232 in provider checkpoint 2024-03-19 01:14:06 proc_lsst.multi:201 [INFO] cancelling 17226232 on provider checkpoint proc_lsst.multi INFO: cancelling 17226232 on provider checkpoint 2024-03-19 01:14:06 proc_lsst.multi:146 [INFO] found job 17226233 in provider checkpoint proc_lsst.multi INFO: found job 17226233 in provider checkpoint 2024-03-19 01:14:06 proc_lsst.multi:201 [INFO] cancelling 17226233 on provider checkpoint proc_lsst.multi INFO: cancelling 17226233 on provider checkpoint 2024-03-19 01:14:07 proc_lsst.multi:146 [INFO] found job 17226237 in provider checkpoint proc_lsst.multi INFO: found job 17226237 in provider checkpoint 2024-03-19 01:14:07 proc_lsst.multi:201 [INFO] cancelling 17226237 on provider checkpoint proc_lsst.multi INFO: cancelling 17226237 on provider checkpoint 2024-03-19 01:14:07 proc_lsst.multi:146 [INFO] found job 17226238 in provider checkpoint proc_lsst.multi INFO: found job 17226238 in provider checkpoint 2024-03-19 01:14:07 proc_lsst.multi:201 [INFO] cancelling 17226238 on provider checkpoint proc_lsst.multi INFO: cancelling 17226238 on provider checkpoint 2024-03-19 01:14:07 proc_lsst.multi:146 [INFO] found job 17226175 in provider astro proc_lsst.multi INFO: found job 17226175 in provider astro 2024-03-19 01:14:07 proc_lsst.multi:201 [INFO] cancelling 17226175 on provider astro proc_lsst.multi INFO: cancelling 17226175 on provider astro 2024-03-19 01:14:07 proc_lsst.multi:146 [INFO] found job 17226176 in provider astro proc_lsst.multi INFO: found job 17226176 in provider astro 2024-03-19 01:14:07 proc_lsst.multi:201 [INFO] cancelling 17226176 on provider astro proc_lsst.multi INFO: cancelling 17226176 on provider astro 2024-03-19 01:14:07 proc_lsst.multi:146 [INFO] found job 17226177 in provider astro proc_lsst.multi INFO: found job 17226177 in provider astro 2024-03-19 01:14:07 proc_lsst.multi:201 [INFO] cancelling 17226177 on provider astro proc_lsst.multi INFO: cancelling 17226177 on provider astro 2024-03-19 01:14:07 proc_lsst.multi:146 [INFO] found job 17226178 in provider astro proc_lsst.multi INFO: found job 17226178 in provider astro 2024-03-19 01:14:07 proc_lsst.multi:201 [INFO] cancelling 17226178 on provider astro proc_lsst.multi INFO: cancelling 17226178 on provider astro 2024-03-19 01:14:07 proc_lsst.multi:146 [INFO] found job 17226179 in provider astro proc_lsst.multi INFO: found job 17226179 in provider astro 2024-03-19 01:14:07 proc_lsst.multi:201 [INFO] cancelling 17226179 on provider astro proc_lsst.multi INFO: cancelling 17226179 on provider astro 2024-03-19 01:14:07 proc_lsst.multi:146 [INFO] found job 17226180 in provider astro proc_lsst.multi INFO: found job 17226180 in provider astro 2024-03-19 01:14:07 proc_lsst.multi:201 [INFO] cancelling 17226180 on provider astro proc_lsst.multi INFO: cancelling 17226180 on provider astro 2024-03-19 01:14:07 proc_lsst.multi:146 [INFO] found job 17226181 in provider astro proc_lsst.multi INFO: found job 17226181 in provider astro 2024-03-19 01:14:07 proc_lsst.multi:201 [INFO] cancelling 17226181 on provider astro proc_lsst.multi INFO: cancelling 17226181 on provider astro 2024-03-19 01:14:07 proc_lsst.multi:146 [INFO] found job 17226182 in provider astro proc_lsst.multi INFO: found job 17226182 in provider astro 2024-03-19 01:14:07 proc_lsst.multi:201 [INFO] cancelling 17226182 on provider astro proc_lsst.multi INFO: cancelling 17226182 on provider astro 2024-03-19 01:14:08 proc_lsst.multi:146 [INFO] found job 17226183 in provider astro proc_lsst.multi INFO: found job 17226183 in provider astro 2024-03-19 01:14:08 proc_lsst.multi:201 [INFO] cancelling 17226183 on provider astro proc_lsst.multi INFO: cancelling 17226183 on provider astro 2024-03-19 01:14:08 proc_lsst.multi:146 [INFO] found job 17226184 in provider astro proc_lsst.multi INFO: found job 17226184 in provider astro 2024-03-19 01:14:08 proc_lsst.multi:201 [INFO] cancelling 17226184 on provider astro proc_lsst.multi INFO: cancelling 17226184 on provider astro 2024-03-19 01:14:08 proc_lsst.multi:146 [INFO] found job 17226185 in provider astro proc_lsst.multi INFO: found job 17226185 in provider astro 2024-03-19 01:14:08 proc_lsst.multi:201 [INFO] cancelling 17226185 on provider astro proc_lsst.multi INFO: cancelling 17226185 on provider astro 2024-03-19 01:14:08 proc_lsst.multi:146 [INFO] found job 17226186 in provider astro proc_lsst.multi INFO: found job 17226186 in provider astro 2024-03-19 01:14:08 proc_lsst.multi:201 [INFO] cancelling 17226186 on provider astro proc_lsst.multi INFO: cancelling 17226186 on provider astro 2024-03-19 01:14:08 proc_lsst.multi:146 [INFO] found job 17226187 in provider astro proc_lsst.multi INFO: found job 17226187 in provider astro 2024-03-19 01:14:08 proc_lsst.multi:201 [INFO] cancelling 17226187 on provider astro proc_lsst.multi INFO: cancelling 17226187 on provider astro 2024-03-19 01:14:08 proc_lsst.multi:146 [INFO] found job 17226188 in provider astro proc_lsst.multi INFO: found job 17226188 in provider astro 2024-03-19 01:14:08 proc_lsst.multi:201 [INFO] cancelling 17226188 on provider astro proc_lsst.multi INFO: cancelling 17226188 on provider astro 2024-03-19 01:14:08 proc_lsst.multi:146 [INFO] found job 17226189 in provider astro proc_lsst.multi INFO: found job 17226189 in provider astro 2024-03-19 01:14:08 proc_lsst.multi:201 [INFO] cancelling 17226189 on provider astro proc_lsst.multi INFO: cancelling 17226189 on provider astro 2024-03-19 01:14:08 proc_lsst.multi:146 [INFO] found job 17226190 in provider astro proc_lsst.multi INFO: found job 17226190 in provider astro 2024-03-19 01:14:08 proc_lsst.multi:201 [INFO] cancelling 17226190 on provider astro proc_lsst.multi INFO: cancelling 17226190 on provider astro 2024-03-19 01:14:08 proc_lsst.multi:146 [INFO] found job 17226191 in provider astro proc_lsst.multi INFO: found job 17226191 in provider astro 2024-03-19 01:14:08 proc_lsst.multi:201 [INFO] cancelling 17226191 on provider astro proc_lsst.multi INFO: cancelling 17226191 on provider astro 2024-03-19 01:14:09 proc_lsst.multi:146 [INFO] found job 17226192 in provider astro proc_lsst.multi INFO: found job 17226192 in provider astro 2024-03-19 01:14:09 proc_lsst.multi:201 [INFO] cancelling 17226192 on provider astro proc_lsst.multi INFO: cancelling 17226192 on provider astro 2024-03-19 01:14:09 proc_lsst.multi:146 [INFO] found job 17226193 in provider astro proc_lsst.multi INFO: found job 17226193 in provider astro 2024-03-19 01:14:09 proc_lsst.multi:201 [INFO] cancelling 17226193 on provider astro proc_lsst.multi INFO: cancelling 17226193 on provider astro 2024-03-19 01:14:09 proc_lsst.multi:146 [INFO] found job 17226194 in provider astro proc_lsst.multi INFO: found job 17226194 in provider astro 2024-03-19 01:14:09 proc_lsst.multi:201 [INFO] cancelling 17226194 on provider astro proc_lsst.multi INFO: cancelling 17226194 on provider astro 2024-03-19 01:14:09 proc_lsst.multi:146 [INFO] found job 17226195 in provider astro proc_lsst.multi INFO: found job 17226195 in provider astro 2024-03-19 01:14:09 proc_lsst.multi:201 [INFO] cancelling 17226195 on provider astro proc_lsst.multi INFO: cancelling 17226195 on provider astro 2024-03-19 01:14:09 proc_lsst.multi:146 [INFO] found job 17226196 in provider astro proc_lsst.multi INFO: found job 17226196 in provider astro 2024-03-19 01:14:09 proc_lsst.multi:201 [INFO] cancelling 17226196 on provider astro proc_lsst.multi INFO: cancelling 17226196 on provider astro 2024-03-19 01:14:09 proc_lsst.multi:146 [INFO] found job 17226197 in provider astro proc_lsst.multi INFO: found job 17226197 in provider astro 2024-03-19 01:14:09 proc_lsst.multi:201 [INFO] cancelling 17226197 on provider astro proc_lsst.multi INFO: cancelling 17226197 on provider astro 2024-03-19 01:14:09 proc_lsst.multi:146 [INFO] found job 17226198 in provider astro proc_lsst.multi INFO: found job 17226198 in provider astro 2024-03-19 01:14:09 proc_lsst.multi:201 [INFO] cancelling 17226198 on provider astro proc_lsst.multi INFO: cancelling 17226198 on provider astro 2024-03-19 01:14:09 proc_lsst.multi:146 [INFO] found job 17226199 in provider astro proc_lsst.multi INFO: found job 17226199 in provider astro 2024-03-19 01:14:09 proc_lsst.multi:201 [INFO] cancelling 17226199 on provider astro proc_lsst.multi INFO: cancelling 17226199 on provider astro 2024-03-19 01:14:10 proc_lsst.multi:146 [INFO] found job 17226200 in provider astro proc_lsst.multi INFO: found job 17226200 in provider astro 2024-03-19 01:14:10 proc_lsst.multi:201 [INFO] cancelling 17226200 on provider astro proc_lsst.multi INFO: cancelling 17226200 on provider astro 2024-03-19 01:14:10 proc_lsst.multi:146 [INFO] found job 17226201 in provider astro proc_lsst.multi INFO: found job 17226201 in provider astro 2024-03-19 01:14:10 proc_lsst.multi:201 [INFO] cancelling 17226201 on provider astro proc_lsst.multi INFO: cancelling 17226201 on provider astro 2024-03-19 01:14:10 proc_lsst.multi:146 [INFO] found job 17226202 in provider astro proc_lsst.multi INFO: found job 17226202 in provider astro 2024-03-19 01:14:10 proc_lsst.multi:201 [INFO] cancelling 17226202 on provider astro proc_lsst.multi INFO: cancelling 17226202 on provider astro 2024-03-19 01:14:10 proc_lsst.multi:146 [INFO] found job 17226203 in provider astro proc_lsst.multi INFO: found job 17226203 in provider astro 2024-03-19 01:14:10 proc_lsst.multi:201 [INFO] cancelling 17226203 on provider astro proc_lsst.multi INFO: cancelling 17226203 on provider astro 2024-03-19 01:14:10 proc_lsst.multi:146 [INFO] found job 17226204 in provider astro proc_lsst.multi INFO: found job 17226204 in provider astro 2024-03-19 01:14:10 proc_lsst.multi:201 [INFO] cancelling 17226204 on provider astro proc_lsst.multi INFO: cancelling 17226204 on provider astro 2024-03-19 01:14:10 proc_lsst.multi:146 [INFO] found job 17226210 in provider escience_a40 proc_lsst.multi INFO: found job 17226210 in provider escience_a40 2024-03-19 01:14:10 proc_lsst.multi:201 [INFO] cancelling 17226210 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226210 on provider escience_a40 2024-03-19 01:14:10 proc_lsst.multi:146 [INFO] found job 17226211 in provider escience_a40 proc_lsst.multi INFO: found job 17226211 in provider escience_a40 2024-03-19 01:14:10 proc_lsst.multi:201 [INFO] cancelling 17226211 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226211 on provider escience_a40 2024-03-19 01:14:10 proc_lsst.multi:146 [INFO] found job 17226212 in provider escience_a40 proc_lsst.multi INFO: found job 17226212 in provider escience_a40 2024-03-19 01:14:10 proc_lsst.multi:201 [INFO] cancelling 17226212 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226212 on provider escience_a40 2024-03-19 01:14:10 proc_lsst.multi:146 [INFO] found job 17226213 in provider escience_a40 proc_lsst.multi INFO: found job 17226213 in provider escience_a40 2024-03-19 01:14:10 proc_lsst.multi:201 [INFO] cancelling 17226213 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226213 on provider escience_a40 2024-03-19 01:14:10 proc_lsst.multi:146 [INFO] found job 17226214 in provider escience_a40 proc_lsst.multi INFO: found job 17226214 in provider escience_a40 2024-03-19 01:14:10 proc_lsst.multi:201 [INFO] cancelling 17226214 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226214 on provider escience_a40 2024-03-19 01:14:10 proc_lsst.multi:146 [INFO] found job 17226215 in provider escience_a40 proc_lsst.multi INFO: found job 17226215 in provider escience_a40 2024-03-19 01:14:10 proc_lsst.multi:201 [INFO] cancelling 17226215 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226215 on provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:146 [INFO] found job 17226216 in provider escience_a40 proc_lsst.multi INFO: found job 17226216 in provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:201 [INFO] cancelling 17226216 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226216 on provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:146 [INFO] found job 17226217 in provider escience_a40 proc_lsst.multi INFO: found job 17226217 in provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:201 [INFO] cancelling 17226217 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226217 on provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:146 [INFO] found job 17226218 in provider escience_a40 proc_lsst.multi INFO: found job 17226218 in provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:201 [INFO] cancelling 17226218 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226218 on provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:146 [INFO] found job 17226219 in provider escience_a40 proc_lsst.multi INFO: found job 17226219 in provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:201 [INFO] cancelling 17226219 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226219 on provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:146 [INFO] found job 17226220 in provider escience_a40 proc_lsst.multi INFO: found job 17226220 in provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:201 [INFO] cancelling 17226220 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226220 on provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:146 [INFO] found job 17226221 in provider escience_a40 proc_lsst.multi INFO: found job 17226221 in provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:201 [INFO] cancelling 17226221 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226221 on provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:146 [INFO] found job 17226222 in provider escience_a40 proc_lsst.multi INFO: found job 17226222 in provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:201 [INFO] cancelling 17226222 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226222 on provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:146 [INFO] found job 17226223 in provider escience_a40 proc_lsst.multi INFO: found job 17226223 in provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:201 [INFO] cancelling 17226223 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226223 on provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:146 [INFO] found job 17226224 in provider escience_a40 proc_lsst.multi INFO: found job 17226224 in provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:201 [INFO] cancelling 17226224 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226224 on provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:146 [INFO] found job 17226225 in provider escience_a40 proc_lsst.multi INFO: found job 17226225 in provider escience_a40 2024-03-19 01:14:11 proc_lsst.multi:201 [INFO] cancelling 17226225 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226225 on provider escience_a40 2024-03-19 01:14:12 proc_lsst.multi:146 [INFO] found job 17226226 in provider escience_a40 proc_lsst.multi INFO: found job 17226226 in provider escience_a40 2024-03-19 01:14:12 proc_lsst.multi:201 [INFO] cancelling 17226226 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226226 on provider escience_a40 2024-03-19 01:14:12 proc_lsst.multi:146 [INFO] found job 17226227 in provider escience_a40 proc_lsst.multi INFO: found job 17226227 in provider escience_a40 2024-03-19 01:14:12 proc_lsst.multi:201 [INFO] cancelling 17226227 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226227 on provider escience_a40 2024-03-19 01:14:12 proc_lsst.multi:146 [INFO] found job 17226228 in provider escience_a40 proc_lsst.multi INFO: found job 17226228 in provider escience_a40 2024-03-19 01:14:12 proc_lsst.multi:201 [INFO] cancelling 17226228 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226228 on provider escience_a40 2024-03-19 01:14:12 proc_lsst.multi:146 [INFO] found job 17226229 in provider escience_a40 proc_lsst.multi INFO: found job 17226229 in provider escience_a40 2024-03-19 01:14:12 proc_lsst.multi:201 [INFO] cancelling 17226229 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226229 on provider escience_a40 2024-03-19 01:14:12 proc_lsst.multi:146 [INFO] found job 17226205 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226205 in provider escience_rtx6k 2024-03-19 01:14:12 proc_lsst.multi:201 [INFO] cancelling 17226205 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17226205 on provider escience_rtx6k 2024-03-19 01:14:12 proc_lsst.multi:146 [INFO] found job 17226206 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226206 in provider escience_rtx6k 2024-03-19 01:14:12 proc_lsst.multi:201 [INFO] cancelling 17226206 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17226206 on provider escience_rtx6k 2024-03-19 01:14:12 proc_lsst.multi:146 [INFO] found job 17226207 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226207 in provider escience_rtx6k 2024-03-19 01:14:12 proc_lsst.multi:201 [INFO] cancelling 17226207 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17226207 on provider escience_rtx6k 2024-03-19 01:14:12 proc_lsst.multi:146 [INFO] found job 17226208 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226208 in provider escience_rtx6k 2024-03-19 01:14:12 proc_lsst.multi:201 [INFO] cancelling 17226208 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17226208 on provider escience_rtx6k 2024-03-19 01:14:12 proc_lsst.multi:146 [INFO] found job 17226209 in provider escience_rtx6k proc_lsst.multi INFO: found job 17226209 in provider escience_rtx6k 2024-03-19 01:14:12 proc_lsst.multi:201 [INFO] cancelling 17226209 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17226209 on provider escience_rtx6k 2024-03-19 01:14:12 proc_lsst.multi:146 [INFO] found job 25873 in provider local proc_lsst.multi INFO: found job 25873 in provider local 2024-03-19 01:14:12 proc_lsst.multi:201 [INFO] cancelling 25873 on provider local proc_lsst.multi INFO: cancelling 25873 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 25873 parsl.providers.local.local WARNING: Failed to kill PID: 25873 and child processes on local 2024-03-19 01:14:13 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 803.7065 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20220527_A0c_science#step1_20240319T010033Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 803.7132 seconds; current memory usage: 0.372 Gibyte, delta: 0.007 Gibyte, peak delta: 0.007 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 823.0765 seconds; current memory usage: 0.372 Gibyte, delta: 0.235 Gibyte, peak delta: 0.235 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.372 Gibyte (main), 0.372 Gibyte (largest child process) Run Id: None Run Name: DEEP_20220527_A0c_science#step1_20240319T010033Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 13m45.361s user 1m45.891s sys 0m59.513s