lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 13.0574 seconds; current memory usage: 0.179 Gibyte, delta: 0.043 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0039 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0085 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/pipeline_orig.qgraph DEEP/20220822/B1m/science#step0/20240307T062031Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.5062 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 6.4960 seconds; current memory usage: 0.269 Gibyte, delta: 0.090 Gibyte, peak delta: 0.079 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.269 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0126 seconds; current memory usage: 0.269 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.269 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 124 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20220822_B1m_science#step0_20240307T062031Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0365 seconds; current memory usage: 0.269 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.269 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 126 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.1670 seconds; current memory usage: 0.270 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 0a98b6bf-e95b-4414-b3b5-f0e2387fc3d3 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-07 06:21:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-07 06:21:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 06:21:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-07 06:21:02 proc_lsst.multi:170 [INFO] [multi] job_id 64667 proc_lsst.multi INFO: [multi] job_id 64667 2024-03-07 06:21:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:02 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 64667 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 64667 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 64667 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 032e559ff9a0fa76680d3a69d6929a35 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56ad87f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/6c5cad59-d235-4b9a-81ec-d6d0f0468d11_isrForCrosstalkSources_1124443_21.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/6c5cad59-d235-4b9a-81ec-d6d0f0468d11_isrForCrosstalkSources_1124443_21.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 39de1270fc12a2eff70050de43cb57cd parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/f50e8f63-e152-4115-83c6-040833451b18_isrForCrosstalkSources_1124443_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/f50e8f63-e152-4115-83c6-040833451b18_isrForCrosstalkSources_1124443_48.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 3d364a127bf687bf13fbb0f1813db42d parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/f35ac62f-b778-4cd3-87fb-2e4440c8f32a_isrForCrosstalkSources_1124445_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/f35ac62f-b778-4cd3-87fb-2e4440c8f32a_isrForCrosstalkSources_1124445_37.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 3242ab9b2fe7761e1acbfd3d2c8ffa4c parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/5eb11b83-2f14-4beb-9380-dd9e765b1493_isrForCrosstalkSources_1124443_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/5eb11b83-2f14-4beb-9380-dd9e765b1493_isrForCrosstalkSources_1124443_4.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 17e1add9c3a6f4093e674ebdc97cd987 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/187eae1f-ef3d-491a-8fcf-0cd225c1f931_isrForCrosstalkSources_1124443_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/187eae1f-ef3d-491a-8fcf-0cd225c1f931_isrForCrosstalkSources_1124443_34.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 7dd8016441969cf25002cdedc03c9fe0 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/42908c6e-4f48-47f7-8fb6-933ca0c0fe40_isrForCrosstalkSources_1124443_43.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/42908c6e-4f48-47f7-8fb6-933ca0c0fe40_isrForCrosstalkSources_1124443_43.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash dcdecb3d48036982f04c746e7dd4862a parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56ad87920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/631186c9-70ef-4f02-93f2-b7962edb5be1_isrForCrosstalkSources_1124443_22.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/631186c9-70ef-4f02-93f2-b7962edb5be1_isrForCrosstalkSources_1124443_22.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 1abd8e5788849ce6c0298fdabcad95b8 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5603b68f-e7d0-43c7-b1c1-ad32d6168dbc_isrForCrosstalkSources_1124445_18.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5603b68f-e7d0-43c7-b1c1-ad32d6168dbc_isrForCrosstalkSources_1124445_18.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 70f26a2580b4ac18ebb1331149e805cc parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/baffd39e-71ba-4c5f-9d0b-79f226e026e4_isrForCrosstalkSources_1124445_17.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/baffd39e-71ba-4c5f-9d0b-79f226e026e4_isrForCrosstalkSources_1124445_17.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 82d3f2ac4622487a009646e7833181c8 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/ad4cc67c-4a55-4e95-bdc7-0f14fff8ed39_isrForCrosstalkSources_1124445_5.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/ad4cc67c-4a55-4e95-bdc7-0f14fff8ed39_isrForCrosstalkSources_1124445_5.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash f50fff995e994c05ddaca3b3101a4196 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/573dba41-7a4f-4fa1-b91a-d62154d3a2f4_isrForCrosstalkSources_1124445_35.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/573dba41-7a4f-4fa1-b91a-d62154d3a2f4_isrForCrosstalkSources_1124445_35.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash a6d58839fa4e00af5d4873cc43c357dc parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/0fcbc47c-0d21-4af9-b3d4-60e669bb3e4a_isrForCrosstalkSources_1124443_3.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/0fcbc47c-0d21-4af9-b3d4-60e669bb3e4a_isrForCrosstalkSources_1124443_3.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash f53b5756e578c7410d1e90424711e813 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/7824f045-ab10-4cc4-b8e0-628c833c97d9_isrForCrosstalkSources_1124445_13.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/7824f045-ab10-4cc4-b8e0-628c833c97d9_isrForCrosstalkSources_1124445_13.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 578d6d5c32fecebde8b659ab2c472c09 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/ec7e34fe-1afe-459f-b8ee-1f2dc49a78e5_isrForCrosstalkSources_1124445_58.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/ec7e34fe-1afe-459f-b8ee-1f2dc49a78e5_isrForCrosstalkSources_1124445_58.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 5ff478d0cbe5240340a8f7385a05a330 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/9c5534dd-a4ba-4017-87c8-7947fcce6b3d_isrForCrosstalkSources_1124445_24.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/9c5534dd-a4ba-4017-87c8-7947fcce6b3d_isrForCrosstalkSources_1124445_24.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 319bb3d106ac123219fdd4f79fc71961 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/d4601d22-fc42-4ff5-8ebd-3a99ad740440_isrForCrosstalkSources_1124445_39.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/d4601d22-fc42-4ff5-8ebd-3a99ad740440_isrForCrosstalkSources_1124445_39.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 9fcb35596bf2960ed6b82f302feed1e7 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/facfcad1-7a6a-4d03-9ea3-7c1330cec9c9_isrForCrosstalkSources_1124443_33.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/facfcad1-7a6a-4d03-9ea3-7c1330cec9c9_isrForCrosstalkSources_1124443_33.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash c1b3f1f437f1f7dffb6e86a13d429ff4 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ef81bc20-8199-4bdb-9ddf-8329a332d10e_isrForCrosstalkSources_1124443_6.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ef81bc20-8199-4bdb-9ddf-8329a332d10e_isrForCrosstalkSources_1124443_6.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 67bd8fb0c03383a5b0e1879749ccd2bb parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/22f9d3a8-57e3-4be8-ae29-828bc999ccda_isrForCrosstalkSources_1124443_11.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/22f9d3a8-57e3-4be8-ae29-828bc999ccda_isrForCrosstalkSources_1124443_11.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 0612fc94e7eaa158f003dcd6bc00930a parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/97d3b131-8eaf-4d14-ba3c-6ef9e3f61d60_isrForCrosstalkSources_1124443_54.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/97d3b131-8eaf-4d14-ba3c-6ef9e3f61d60_isrForCrosstalkSources_1124443_54.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash e432cfa9d2e4b7a46150bd4ec4e389d6 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/151bb07d-d456-4ddd-89cc-6285b2f3993b_isrForCrosstalkSources_1124445_51.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/151bb07d-d456-4ddd-89cc-6285b2f3993b_isrForCrosstalkSources_1124445_51.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 55c5a679b799346a5e0301bd21a69110 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/be1928fa-a0e6-45f5-b8fa-8c22baf6a252_isrForCrosstalkSources_1124443_35.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/be1928fa-a0e6-45f5-b8fa-8c22baf6a252_isrForCrosstalkSources_1124443_35.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash aa3f9cc375db67336bb157dbbd90f29d parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/972a98e9-7434-46a7-97a4-6f9c3eeda4d0_isrForCrosstalkSources_1124445_6.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/972a98e9-7434-46a7-97a4-6f9c3eeda4d0_isrForCrosstalkSources_1124445_6.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash cde0f1573866d31f7d9ed12af77bfe30 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/caa52a88-13b1-4b52-8d32-59e66fbf2f70_isrForCrosstalkSources_1124445_41.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/caa52a88-13b1-4b52-8d32-59e66fbf2f70_isrForCrosstalkSources_1124445_41.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 89f733aefdd855770197a17af57f9991 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/6ecdce67-5018-4dab-9b3d-4fdd52b25ac1_isrForCrosstalkSources_1124445_43.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/6ecdce67-5018-4dab-9b3d-4fdd52b25ac1_isrForCrosstalkSources_1124445_43.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash ef66adc19e165caa1341e336714244f4 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/8922fb0f-d524-48d4-a4c7-ac1e7fbb2778_isrForCrosstalkSources_1124443_44.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/8922fb0f-d524-48d4-a4c7-ac1e7fbb2778_isrForCrosstalkSources_1124443_44.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 660ef2066d6f4877e3f0e0feb391908d parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/39d54a05-e190-4b10-98ba-8411cb4f81b6_isrForCrosstalkSources_1124443_13.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/39d54a05-e190-4b10-98ba-8411cb4f81b6_isrForCrosstalkSources_1124443_13.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash f2cb666202a2c188f043381467c710c8 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/34239b95-ccdc-4165-8e63-7db478591913_isrForCrosstalkSources_1124443_49.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/34239b95-ccdc-4165-8e63-7db478591913_isrForCrosstalkSources_1124443_49.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 9c38d589dd5e7d9fe49299cf17e8a395 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/fb01d323-ba6d-4eba-89cc-f6340d4ca044_isrForCrosstalkSources_1124445_57.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/fb01d323-ba6d-4eba-89cc-f6340d4ca044_isrForCrosstalkSources_1124445_57.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash e45326dbb7accae9d8b49ddd08e990eb parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ef07ffc4-96e6-4603-88e5-860641af4e08_isrForCrosstalkSources_1124443_42.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ef07ffc4-96e6-4603-88e5-860641af4e08_isrForCrosstalkSources_1124443_42.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 38a9986153abc1b2e94baf44c0c31d31 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/f89fdf8a-4fc4-4dc1-aab9-2555bfff5816_isrForCrosstalkSources_1124445_50.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/f89fdf8a-4fc4-4dc1-aab9-2555bfff5816_isrForCrosstalkSources_1124445_50.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 77b074f2b65cb0a2e82c8a4a8a3d9789 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/d2f56516-9d49-4c12-b53c-3658c833ab9c_isrForCrosstalkSources_1124445_32.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/d2f56516-9d49-4c12-b53c-3658c833ab9c_isrForCrosstalkSources_1124445_32.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 55ae34fe13786356c6260c680fa45d76 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/f8796eae-3f92-4a13-8f76-ee80b70a1f8c_isrForCrosstalkSources_1124445_46.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/f8796eae-3f92-4a13-8f76-ee80b70a1f8c_isrForCrosstalkSources_1124445_46.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 825c9919b4b71a3cb48a94af6d784ae2 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/b0fa308c-561b-4720-9efa-9df51e57db3b_isrForCrosstalkSources_1124445_4.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/b0fa308c-561b-4720-9efa-9df51e57db3b_isrForCrosstalkSources_1124445_4.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 3493c988a7a6c32ec7eecd36667525f9 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/61630073-3ab3-43b0-81ea-5e468a2d3ed6_isrForCrosstalkSources_1124443_14.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/61630073-3ab3-43b0-81ea-5e468a2d3ed6_isrForCrosstalkSources_1124443_14.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash cd3fc5e338269a8867b91110e475479c parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b1e2385c-343e-441f-92e2-34fceb67d14b_isrForCrosstalkSources_1124443_25.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b1e2385c-343e-441f-92e2-34fceb67d14b_isrForCrosstalkSources_1124443_25.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 3085c9d276bee28629fd7f5acccd3b97 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/98ac31c1-f5d2-4c52-9172-522e57c73bbc_isrForCrosstalkSources_1124445_27.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/98ac31c1-f5d2-4c52-9172-522e57c73bbc_isrForCrosstalkSources_1124445_27.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 5e6743c72ef43ddca57b43b7581a3c23 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/4b0e774d-447b-4320-a39e-c11f2b69f361_isrForCrosstalkSources_1124445_8.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/4b0e774d-447b-4320-a39e-c11f2b69f361_isrForCrosstalkSources_1124445_8.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 3809dd18c7b67c84c2c8230d91f65a0d parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/6d9a116f-3dc2-4aa8-871d-955820682d1a_isrForCrosstalkSources_1124443_19.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/6d9a116f-3dc2-4aa8-871d-955820682d1a_isrForCrosstalkSources_1124443_19.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash ff49d46491fa1ad086831d467a865ea0 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/be32d4cf-77a6-4d0d-9ade-aef35fd1d7e0_isrForCrosstalkSources_1124443_31.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/be32d4cf-77a6-4d0d-9ade-aef35fd1d7e0_isrForCrosstalkSources_1124443_31.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash c6b47f4c63a56027920a03007ada431a parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/908519c7-5afc-4ea9-95ea-13d96820f911_isrForCrosstalkSources_1124445_15.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/908519c7-5afc-4ea9-95ea-13d96820f911_isrForCrosstalkSources_1124445_15.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 31b060146f6cc2c2f05c6cb0be15a90e parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/a66797ff-98f3-469f-af3a-f138162ae55d_isrForCrosstalkSources_1124445_19.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/a66797ff-98f3-469f-af3a-f138162ae55d_isrForCrosstalkSources_1124445_19.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 6b404a4d2946e96886e2fe2fedd5d968 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/a88ed264-1c32-4eac-a274-a79b8a7d84f5_isrForCrosstalkSources_1124443_36.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/a88ed264-1c32-4eac-a274-a79b8a7d84f5_isrForCrosstalkSources_1124443_36.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 7d22d66e3fea606a0b8310395db3e543 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/6d52ec1f-f461-4bd9-bf15-e8504b0cbf9b_isrForCrosstalkSources_1124445_42.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/6d52ec1f-f461-4bd9-bf15-e8504b0cbf9b_isrForCrosstalkSources_1124445_42.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash f2b33e5f1550c4487b446326986f4b36 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/9da4cdc1-5502-4103-b045-cc480022be43_isrForCrosstalkSources_1124445_48.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/9da4cdc1-5502-4103-b045-cc480022be43_isrForCrosstalkSources_1124445_48.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 3bb331a77a91c1af8e698747a05500df parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/2eedaa37-2225-4341-b4fa-9be50ca5f86e_isrForCrosstalkSources_1124445_20.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/2eedaa37-2225-4341-b4fa-9be50ca5f86e_isrForCrosstalkSources_1124445_20.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 54c5d1d22b88a43820176b3bb0c7fc41 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/54ffa00b-b643-4177-8340-06eaeb34e6b5_isrForCrosstalkSources_1124445_44.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/54ffa00b-b643-4177-8340-06eaeb34e6b5_isrForCrosstalkSources_1124445_44.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 1e1fa682b6d11328fb8982c991239d65 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/bcfaeb96-ffbd-45b9-871e-21ebf2c4f28b_isrForCrosstalkSources_1124445_9.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/bcfaeb96-ffbd-45b9-871e-21ebf2c4f28b_isrForCrosstalkSources_1124445_9.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash be9bde870b5d03414b337bee95dbe2e6 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ba0c7231-c869-495b-94fd-dbfaf193857e_isrForCrosstalkSources_1124443_32.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ba0c7231-c869-495b-94fd-dbfaf193857e_isrForCrosstalkSources_1124443_32.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 285982ae19e046ca686d65ec22ab735f parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/8d1dbbd4-7694-4e0d-acbf-9e058f3d8ffe_isrForCrosstalkSources_1124443_28.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/8d1dbbd4-7694-4e0d-acbf-9e058f3d8ffe_isrForCrosstalkSources_1124443_28.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 61a5b8a0f4d403f6a1c8757b5d987ecc parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/5be3a009-cfce-4673-9a8d-aee1fc5730dc_isrForCrosstalkSources_1124443_5.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/5be3a009-cfce-4673-9a8d-aee1fc5730dc_isrForCrosstalkSources_1124443_5.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash f0f3f5967ff3994256bee7ec1039da28 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/c3991f15-8e1a-4f10-9c3b-bedfa4a81de4_isrForCrosstalkSources_1124443_53.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/c3991f15-8e1a-4f10-9c3b-bedfa4a81de4_isrForCrosstalkSources_1124443_53.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 7cd8bdc2ed741a1efe6ce17f654f4e59 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5f0e19af-0add-440a-a4c0-47b526b9c62f_isrForCrosstalkSources_1124445_23.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5f0e19af-0add-440a-a4c0-47b526b9c62f_isrForCrosstalkSources_1124445_23.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 34800269173fbcd3f7e658f58cad8e3e parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/b537f6a3-1766-4851-9c77-0a1e39d7a9e4_isrForCrosstalkSources_1124445_28.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/b537f6a3-1766-4851-9c77-0a1e39d7a9e4_isrForCrosstalkSources_1124445_28.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 675d81a164fda6f4bcbac707e4f70885 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b07d4dc4-72d1-4c27-93cb-b35426abe73f_isrForCrosstalkSources_1124443_41.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b07d4dc4-72d1-4c27-93cb-b35426abe73f_isrForCrosstalkSources_1124443_41.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 0b7282d261348ba60bd579bff707eab6 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/42b994f4-0af5-4eeb-857f-15517d80593f_isrForCrosstalkSources_1124443_15.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/42b994f4-0af5-4eeb-857f-15517d80593f_isrForCrosstalkSources_1124443_15.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 1569076282dd4a0bcd61ec8cb519ab19 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/83c327b0-41d7-4c63-8bb5-a7e0147c3372_isrForCrosstalkSources_1124445_21.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/83c327b0-41d7-4c63-8bb5-a7e0147c3372_isrForCrosstalkSources_1124445_21.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 67fbfaa6890105bd031245b2af76b0a4 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b4e10b32-4a66-4c76-a9e2-74d975bbe6df_isrForCrosstalkSources_1124443_59.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b4e10b32-4a66-4c76-a9e2-74d975bbe6df_isrForCrosstalkSources_1124443_59.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 990a7dec42e63c804a551216366d770c parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/127d89b8-3c56-4121-8b12-3aa325d9cfb6_isrForCrosstalkSources_1124443_29.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/127d89b8-3c56-4121-8b12-3aa325d9cfb6_isrForCrosstalkSources_1124443_29.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 2d2ef17cf30b2b24ebc2b71f8dc4b54f parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/3af8ad10-87e7-4241-9551-bbd327be06ab_isrForCrosstalkSources_1124443_10.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/3af8ad10-87e7-4241-9551-bbd327be06ab_isrForCrosstalkSources_1124443_10.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash c9508e6cffaf6956868c847965a9ef8d parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/de336afa-ca4b-47e6-b8fc-2c2580a4fd5e_isrForCrosstalkSources_1124445_25.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/de336afa-ca4b-47e6-b8fc-2c2580a4fd5e_isrForCrosstalkSources_1124445_25.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 945073ded0f5c911ceb16f2589fac59a parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/7340798f-805a-4821-b640-42b250b1cfa2_isrForCrosstalkSources_1124443_26.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/7340798f-805a-4821-b640-42b250b1cfa2_isrForCrosstalkSources_1124443_26.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 1a1677c94731abf2e176c58a857eb2dd parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/c5d176dc-a5ea-440b-ada0-d6fa638cde42_isrForCrosstalkSources_1124443_27.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/c5d176dc-a5ea-440b-ada0-d6fa638cde42_isrForCrosstalkSources_1124443_27.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 661308e64d5f787a384f3cd533947d88 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/e938f921-4354-4f21-a793-d8f9f3911f8a_isrForCrosstalkSources_1124443_8.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/e938f921-4354-4f21-a793-d8f9f3911f8a_isrForCrosstalkSources_1124443_8.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 8322b581821d18bedb51624e826c1cdd parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/19fae75c-5e44-4453-b8ec-caeb3ff8822e_isrForCrosstalkSources_1124445_33.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/19fae75c-5e44-4453-b8ec-caeb3ff8822e_isrForCrosstalkSources_1124445_33.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 2e88130dc939e79742d634a2685a057f parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/1e41c3b9-cd23-448b-a9d0-6cceb5e9466d_isrForCrosstalkSources_1124443_50.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/1e41c3b9-cd23-448b-a9d0-6cceb5e9466d_isrForCrosstalkSources_1124443_50.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 265210e01fa585c979e8d8ec61d789e0 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ddbc1786-0efd-43ae-a3ad-36c95589cf2b_isrForCrosstalkSources_1124443_16.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ddbc1786-0efd-43ae-a3ad-36c95589cf2b_isrForCrosstalkSources_1124443_16.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 473e50d4af6b3612da442a1e8af7070d parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/258910a1-85bf-409e-88e4-44594455e3de_isrForCrosstalkSources_1124443_38.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/258910a1-85bf-409e-88e4-44594455e3de_isrForCrosstalkSources_1124443_38.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 7c0720dfec9ea3f66fd3ec0ece39a1e8 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/33f177f4-0a1e-4b5c-998a-6985ff55e7eb_isrForCrosstalkSources_1124445_12.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/33f177f4-0a1e-4b5c-998a-6985ff55e7eb_isrForCrosstalkSources_1124445_12.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash c73045d1d25c822c8b16766569dedc03 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/7464598a-88a8-40b2-afe5-34d129242b2f_isrForCrosstalkSources_1124445_11.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/7464598a-88a8-40b2-afe5-34d129242b2f_isrForCrosstalkSources_1124445_11.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash cfa997494e4c1b87a9a433e8878a3e22 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/e55c2f22-20a3-4fae-ba52-b414d993d339_isrForCrosstalkSources_1124443_18.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/e55c2f22-20a3-4fae-ba52-b414d993d339_isrForCrosstalkSources_1124443_18.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 30d4e24b05b9d3025eae2b6d66205e99 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b0512b34-65e4-45da-9f09-6a7ea98e53e9_isrForCrosstalkSources_1124443_56.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b0512b34-65e4-45da-9f09-6a7ea98e53e9_isrForCrosstalkSources_1124443_56.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 249fb32acf1dc4028f821fec4265b6df parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/79e3dbea-2d8c-47b0-9982-54a426e6cd47_isrForCrosstalkSources_1124445_31.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/79e3dbea-2d8c-47b0-9982-54a426e6cd47_isrForCrosstalkSources_1124445_31.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 9b9fe3babd6085d297a6d55a71af9cec parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/f7e72421-82b5-4b15-95bb-5b82a6177216_isrForCrosstalkSources_1124443_46.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/f7e72421-82b5-4b15-95bb-5b82a6177216_isrForCrosstalkSources_1124443_46.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 4369021963ca0479c22d867c64b2fe9d parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/65cc0942-9990-4205-9d14-3c01f7226abb_isrForCrosstalkSources_1124443_30.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/65cc0942-9990-4205-9d14-3c01f7226abb_isrForCrosstalkSources_1124443_30.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 5ec9318a308e95b7519d019f3ac2dd76 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/c8fb8957-71f1-4546-84a4-171ae0019e9e_isrForCrosstalkSources_1124445_14.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/c8fb8957-71f1-4546-84a4-171ae0019e9e_isrForCrosstalkSources_1124445_14.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 9822b701501f640d20db9ec4714fc167 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/0cac22f2-d851-4fec-aa7a-93d67bc8499d_isrForCrosstalkSources_1124443_58.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/0cac22f2-d851-4fec-aa7a-93d67bc8499d_isrForCrosstalkSources_1124443_58.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 3820893921c3b5f5f7075f48fcf92266 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5891b97f-d6e0-4770-ba08-bc0353f7c171_isrForCrosstalkSources_1124445_62.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5891b97f-d6e0-4770-ba08-bc0353f7c171_isrForCrosstalkSources_1124445_62.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash e59f403420aed55a0e37d3c71209df1c parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/56228c9a-4343-4aca-86ec-04ca413b18a4_isrForCrosstalkSources_1124445_52.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/56228c9a-4343-4aca-86ec-04ca413b18a4_isrForCrosstalkSources_1124445_52.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 2791f30002b4099b80c918990fcd6729 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/0201e746-5637-41a5-80b4-911bbd21e7e4_isrForCrosstalkSources_1124445_47.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/0201e746-5637-41a5-80b4-911bbd21e7e4_isrForCrosstalkSources_1124445_47.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash d46d14912fac6a1eee912862007855a3 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/35c85dad-9eb0-43a0-95ac-e708b1602bfe_isrForCrosstalkSources_1124445_55.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/35c85dad-9eb0-43a0-95ac-e708b1602bfe_isrForCrosstalkSources_1124445_55.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash aa3390138ca251fe468f19809d2332b7 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/e08b47fe-aa7d-4bcb-9222-5d3e6c04ca86_isrForCrosstalkSources_1124445_53.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/e08b47fe-aa7d-4bcb-9222-5d3e6c04ca86_isrForCrosstalkSources_1124445_53.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 99c6aea370d421344713df2b6c4ad1d2 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/defa20ce-9562-42bb-b54f-d1c4738a1108_isrForCrosstalkSources_1124445_7.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/defa20ce-9562-42bb-b54f-d1c4738a1108_isrForCrosstalkSources_1124445_7.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash cb7d5f7c19df1956224f0f40727d2e5b parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/8fabb69d-da6c-4623-ba02-4e628d4e0fde_isrForCrosstalkSources_1124443_37.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/8fabb69d-da6c-4623-ba02-4e628d4e0fde_isrForCrosstalkSources_1124443_37.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 7b8914fd6756998cdb1ca7aa291baf65 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/1e69ed03-ca05-4259-8542-ba7752339e53_isrForCrosstalkSources_1124443_1.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/1e69ed03-ca05-4259-8542-ba7752339e53_isrForCrosstalkSources_1124443_1.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 01e977d0fc70692874c298fc173a1a70 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/d9d6097b-5e1a-4899-b2a1-7b0077838223_isrForCrosstalkSources_1124445_3.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/d9d6097b-5e1a-4899-b2a1-7b0077838223_isrForCrosstalkSources_1124445_3.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 785c0a4d7d885a2e045181d404a76f06 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/83c6c691-0da3-4417-b4e6-6ec755ce904c_isrForCrosstalkSources_1124445_30.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/83c6c691-0da3-4417-b4e6-6ec755ce904c_isrForCrosstalkSources_1124445_30.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash bdeaea6f7acd3bf3c12ce0a1f56c48a9 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5c2b0760-14cf-4743-b7e9-9307304c54f6_isrForCrosstalkSources_1124445_16.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5c2b0760-14cf-4743-b7e9-9307304c54f6_isrForCrosstalkSources_1124445_16.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 822f0f3c8a2679a41e3955bc66428f14 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5fdea119-0ba5-45b5-9fd9-ff290e58fd6c_isrForCrosstalkSources_1124445_2.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5fdea119-0ba5-45b5-9fd9-ff290e58fd6c_isrForCrosstalkSources_1124445_2.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash ed5f3f96dc2f4e65f1e77618b0009aad parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/01ca3867-73f3-4ce9-9119-ac23163e6de5_isrForCrosstalkSources_1124445_49.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/01ca3867-73f3-4ce9-9119-ac23163e6de5_isrForCrosstalkSources_1124445_49.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 16d03e22d94d000cd3d7bae983ef73d9 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c56add3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/35359f61-7328-4583-9e8a-98eb2d35bf07_isrForCrosstalkSources_1124445_1.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/35359f61-7328-4583-9e8a-98eb2d35bf07_isrForCrosstalkSources_1124445_1.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash c6820d096596a98b09b6dc0015be6920 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c5698240e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/e078871f-0631-43ff-a81a-dfc8022c24fe_isrForCrosstalkSources_1124445_26.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/e078871f-0631-43ff-a81a-dfc8022c24fe_isrForCrosstalkSources_1124445_26.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 4401b0a47944346d7daf0e86c6f0c3f8 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/4d25282a-5574-4a0f-8549-8bc32f8b02a5_isrForCrosstalkSources_1124445_34.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/4d25282a-5574-4a0f-8549-8bc32f8b02a5_isrForCrosstalkSources_1124445_34.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash c54d90930f18be590bc772291fe18199 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/4a8051d1-dc7b-4e06-a04a-f22b3fcc4ef4_isrForCrosstalkSources_1124445_54.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/4a8051d1-dc7b-4e06-a04a-f22b3fcc4ef4_isrForCrosstalkSources_1124445_54.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 437f23a5d9fe2245aa84492eb0cb08b3 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/2fcb0bd4-128c-429c-853e-0ae88009a255_isrForCrosstalkSources_1124443_51.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/2fcb0bd4-128c-429c-853e-0ae88009a255_isrForCrosstalkSources_1124443_51.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash e1574eeb41c50489e5644ecb499f555a parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c5698242c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/39e26e35-e7f3-4506-9bef-344c869aa704_isrForCrosstalkSources_1124443_23.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/39e26e35-e7f3-4506-9bef-344c869aa704_isrForCrosstalkSources_1124443_23.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash ef2c87aba88e61dc3fba77a8a46bb464 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/fce36c87-ecf5-43e7-999a-b0ab74d881d1_isrForCrosstalkSources_1124443_55.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/fce36c87-ecf5-43e7-999a-b0ab74d881d1_isrForCrosstalkSources_1124443_55.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash b1fb2df058f0c7e3b270cfc6bf9fb112 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c5698244a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/958c3e36-a631-4b27-bb71-9cfd7c10c7f1_isrForCrosstalkSources_1124443_39.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/958c3e36-a631-4b27-bb71-9cfd7c10c7f1_isrForCrosstalkSources_1124443_39.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash c353958b5bd4c0bbfedffeb1eae67858 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/21eb5745-1b85-48d2-a998-3af617e820c2_isrForCrosstalkSources_1124445_61.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/21eb5745-1b85-48d2-a998-3af617e820c2_isrForCrosstalkSources_1124445_61.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 596b8729efc79f7dc8a5eb1efaaa3e76 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/a594a4d3-74dc-4c9d-a88d-5d01fb44aa51_isrForCrosstalkSources_1124443_24.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/a594a4d3-74dc-4c9d-a88d-5d01fb44aa51_isrForCrosstalkSources_1124443_24.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 26c3e06c6e26f4d8c36098c710bcdeb2 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c5698245e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/04c05710-db12-4258-a385-a85dd822baa5_isrForCrosstalkSources_1124443_47.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/04c05710-db12-4258-a385-a85dd822baa5_isrForCrosstalkSources_1124443_47.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash f61c3d52484b87a2165f798958bc8ca6 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/43f7924b-f0d8-47a6-9c0f-ae38c1d71e2a_isrForCrosstalkSources_1124445_36.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/43f7924b-f0d8-47a6-9c0f-ae38c1d71e2a_isrForCrosstalkSources_1124445_36.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 0bb54682f3f5f55ed5ba9888a1c090e2 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c5698247c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/6a63f590-a229-44e2-83d5-5d3425452447_isrForCrosstalkSources_1124445_45.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/6a63f590-a229-44e2-83d5-5d3425452447_isrForCrosstalkSources_1124445_45.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash a850d4ca1e58b12ddac3324e5a33f86d parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/251f6ed5-d86c-40e6-9066-e74ada49ec44_isrForCrosstalkSources_1124445_38.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/251f6ed5-d86c-40e6-9066-e74ada49ec44_isrForCrosstalkSources_1124445_38.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash df18cc4dc8eeb0ec94a323fca042503a parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c5698249a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/5e141566-f4bb-4d7f-8c97-ea805522c579_isrForCrosstalkSources_1124443_2.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/5e141566-f4bb-4d7f-8c97-ea805522c579_isrForCrosstalkSources_1124443_2.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash a5221df3c58467a5dd993ce1bd6987a1 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/2f25774d-6e04-4874-8ee3-142be0c77fe0_isrForCrosstalkSources_1124445_10.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/2f25774d-6e04-4874-8ee3-142be0c77fe0_isrForCrosstalkSources_1124445_10.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 58e9c13bd35e72d21c2799032ecae734 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/53f58e41-5995-4d8b-b303-5faa664fd721_isrForCrosstalkSources_1124443_40.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/53f58e41-5995-4d8b-b303-5faa664fd721_isrForCrosstalkSources_1124443_40.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 7ff053094b397b0afdc144ba8a3faa0f parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/9484492d-d370-4e7b-8b61-53e6ba13c394_isrForCrosstalkSources_1124443_57.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/9484492d-d370-4e7b-8b61-53e6ba13c394_isrForCrosstalkSources_1124443_57.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 48193265e41a4c5fdb047bece403b451 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/809c427d-50a9-4bbe-8880-a0c1584c7583_isrForCrosstalkSources_1124443_60.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/809c427d-50a9-4bbe-8880-a0c1584c7583_isrForCrosstalkSources_1124443_60.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash b6c4b3cce5658f762dc06bbe7493e037 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/6071eb38-f66a-43de-a6f6-acfb4b6c7690_isrForCrosstalkSources_1124443_20.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/6071eb38-f66a-43de-a6f6-acfb4b6c7690_isrForCrosstalkSources_1124443_20.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 3f2ab5f52e7be15c6c6b56909004b62c parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/cb76d97a-d0e3-4d79-82db-71156c3cfba5_isrForCrosstalkSources_1124443_61.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/cb76d97a-d0e3-4d79-82db-71156c3cfba5_isrForCrosstalkSources_1124443_61.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash cd03d0a84cf052552902a3e7d931af95 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/65bda2d7-50d9-4ba2-bf63-53b7995b7880_isrForCrosstalkSources_1124445_60.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/65bda2d7-50d9-4ba2-bf63-53b7995b7880_isrForCrosstalkSources_1124445_60.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 4bbb53a2f5c9e5debda405eee53fea8c parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/61b8365e-f923-44a1-aa0d-f8e4d7d5a08b_isrForCrosstalkSources_1124443_7.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/61b8365e-f923-44a1-aa0d-f8e4d7d5a08b_isrForCrosstalkSources_1124443_7.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 149dd07693ec036fc1c9fa0477d22554 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/8a3504fd-2248-4e80-a363-51c14ee7cf8c_isrForCrosstalkSources_1124445_22.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/8a3504fd-2248-4e80-a363-51c14ee7cf8c_isrForCrosstalkSources_1124445_22.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 5c5253da3d5967ff3c21f4cd3bda39d0 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b50bb89c-ebdf-4c4e-ad7b-cd4ede3dff1c_isrForCrosstalkSources_1124443_62.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b50bb89c-ebdf-4c4e-ad7b-cd4ede3dff1c_isrForCrosstalkSources_1124443_62.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash b9d05ed8ea810c23671e42f17a57bbf3 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569824f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/773c8b3a-7ea9-465e-99f0-6830d81328bc_isrForCrosstalkSources_1124445_56.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/773c8b3a-7ea9-465e-99f0-6830d81328bc_isrForCrosstalkSources_1124445_56.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 60fe4d35129b5864a11c3d4ab74ec965 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569825080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/8e9ef3fa-c5b9-4286-bd90-56c26220e379_isrForCrosstalkSources_1124445_40.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/8e9ef3fa-c5b9-4286-bd90-56c26220e379_isrForCrosstalkSources_1124445_40.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash aab418aa4a82c030cd9253f71a96eb97 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569825120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/4de241f0-5ec2-4485-82cb-350dea762798_isrForCrosstalkSources_1124443_45.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/4de241f0-5ec2-4485-82cb-350dea762798_isrForCrosstalkSources_1124443_45.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 7e96324d167196abc715ef1152a05212 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c5698251c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/96aeef08-a3bc-452e-bf7a-e752dde87c42_isrForCrosstalkSources_1124443_9.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/96aeef08-a3bc-452e-bf7a-e752dde87c42_isrForCrosstalkSources_1124443_9.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 3295d346ad8feec1937f83d9e5264fdb parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569825300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/ed2de177-bf57-4730-90bf-013aab850023_isrForCrosstalkSources_1124445_29.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/ed2de177-bf57-4730-90bf-013aab850023_isrForCrosstalkSources_1124445_29.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash e649f92dcc298c62a852a4859134078f parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569825260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/67ae6411-6ed1-48d9-a3c9-b83ea4153100_isrForCrosstalkSources_1124443_17.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/67ae6411-6ed1-48d9-a3c9-b83ea4153100_isrForCrosstalkSources_1124443_17.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 3e2fac1692ea770aa9ccdf58f485a33e parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c5698253a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/81e78171-8e6b-4f7b-9e5a-eea19c9b39f1_isrForCrosstalkSources_1124443_12.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/81e78171-8e6b-4f7b-9e5a-eea19c9b39f1_isrForCrosstalkSources_1124443_12.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 58c535eb7bdc903bb83571aa41288ea6 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c569825440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/120912f4-1985-4e29-97a1-160c7151b4a7_isrForCrosstalkSources_1124443_52.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/120912f4-1985-4e29-97a1-160c7151b4a7_isrForCrosstalkSources_1124443_52.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 0332368882178ffb8abd186fb4196754 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c5698254e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/49c69afd-775a-43a1-9b51-497b1f7e0cdb_isrForCrosstalkSources_1124445_59.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/49c69afd-775a-43a1-9b51-497b1f7e0cdb_isrForCrosstalkSources_1124445_59.stderr 2024-03-07 06:21:03 proc_lsst.multi:146 [INFO] found job 64667 in provider local proc_lsst.multi INFO: found job 64667 in provider local parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 124 parsl.dataflow.strategy DEBUG: Executor multi has 124 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 30 more blocks parsl.executors.status_handling INFO: Scaling out by 30 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-07 06:21:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-07 06:21:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 06:21:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:04 proc_lsst.multi:170 [INFO] [multi] job_id 16886330 proc_lsst.multi INFO: [multi] job_id 16886330 2024-03-07 06:21:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886330 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886330 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16886330 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-07 06:21:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-07 06:21:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:04 proc_lsst.multi:170 [INFO] [multi] job_id 16886331 proc_lsst.multi INFO: [multi] job_id 16886331 2024-03-07 06:21:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 06:21:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886331 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886331 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16886331 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-07 06:21:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-07 06:21:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 06:21:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:04 proc_lsst.multi:170 [INFO] [multi] job_id 16886332 proc_lsst.multi INFO: [multi] job_id 16886332 2024-03-07 06:21:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 06:21:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886332 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886332 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16886332 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-07 06:21:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-07 06:21:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 06:21:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:04 proc_lsst.multi:170 [INFO] [multi] job_id 16886333 proc_lsst.multi INFO: [multi] job_id 16886333 2024-03-07 06:21:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 06:21:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886333 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886333 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16886333 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-07 06:21:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-07 06:21:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 06:21:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:05 proc_lsst.multi:170 [INFO] [multi] job_id 16886334 proc_lsst.multi INFO: [multi] job_id 16886334 2024-03-07 06:21:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 06:21:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886334 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886334 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16886334 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-07 06:21:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-07 06:21:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 06:21:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:05 proc_lsst.multi:170 [INFO] [multi] job_id 16886335 proc_lsst.multi INFO: [multi] job_id 16886335 2024-03-07 06:21:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 06:21:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886335 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886335 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16886335 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-07 06:21:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-07 06:21:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 06:21:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:05 proc_lsst.multi:170 [INFO] [multi] job_id 16886336 proc_lsst.multi INFO: [multi] job_id 16886336 2024-03-07 06:21:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 06:21:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886336 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886336 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16886336 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-07 06:21:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-07 06:21:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 06:21:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:05 proc_lsst.multi:170 [INFO] [multi] job_id 16886337 proc_lsst.multi INFO: [multi] job_id 16886337 2024-03-07 06:21:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 06:21:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886337 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886337 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16886337 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-07 06:21:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-07 06:21:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 06:21:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:05 proc_lsst.multi:170 [INFO] [multi] job_id 16886338 proc_lsst.multi INFO: [multi] job_id 16886338 2024-03-07 06:21:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 06:21:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886338 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886338 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16886338 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-07 06:21:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-07 06:21:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 06:21:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:06 proc_lsst.multi:170 [INFO] [multi] job_id 16886339 proc_lsst.multi INFO: [multi] job_id 16886339 2024-03-07 06:21:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 06:21:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886339 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886339 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16886339 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-07 06:21:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-07 06:21:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 06:21:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:06 proc_lsst.multi:170 [INFO] [multi] job_id 16886340 proc_lsst.multi INFO: [multi] job_id 16886340 2024-03-07 06:21:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 06:21:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886340 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886340 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16886340 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-07 06:21:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-07 06:21:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 06:21:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:06 proc_lsst.multi:170 [INFO] [multi] job_id 16886341 proc_lsst.multi INFO: [multi] job_id 16886341 2024-03-07 06:21:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 06:21:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886341 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886341 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16886341 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-07 06:21:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-07 06:21:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 06:21:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:07 proc_lsst.multi:170 [INFO] [multi] job_id 16886342 proc_lsst.multi INFO: [multi] job_id 16886342 2024-03-07 06:21:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 06:21:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886342 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886342 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16886342 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-07 06:21:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-07 06:21:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 06:21:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:07 proc_lsst.multi:170 [INFO] [multi] job_id 16886343 proc_lsst.multi INFO: [multi] job_id 16886343 2024-03-07 06:21:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 06:21:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886343 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886343 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16886343 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-07 06:21:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-07 06:21:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 06:21:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:07 proc_lsst.multi:170 [INFO] [multi] job_id 16886344 proc_lsst.multi INFO: [multi] job_id 16886344 2024-03-07 06:21:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 06:21:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886344 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886344 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16886344 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-07 06:21:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-07 06:21:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 06:21:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:07 proc_lsst.multi:170 [INFO] [multi] job_id 16886345 proc_lsst.multi INFO: [multi] job_id 16886345 2024-03-07 06:21:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 06:21:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886345 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886345 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16886345 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-07 06:21:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-07 06:21:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 06:21:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:08 proc_lsst.multi:170 [INFO] [multi] job_id 16886346 proc_lsst.multi INFO: [multi] job_id 16886346 2024-03-07 06:21:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 06:21:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886346 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886346 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16886346 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-07 06:21:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-07 06:21:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 06:21:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:08 proc_lsst.multi:170 [INFO] [multi] job_id 16886347 proc_lsst.multi INFO: [multi] job_id 16886347 2024-03-07 06:21:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 06:21:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886347 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886347 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16886347 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-07 06:21:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-07 06:21:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 06:21:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:08 proc_lsst.multi:170 [INFO] [multi] job_id 16886348 proc_lsst.multi INFO: [multi] job_id 16886348 2024-03-07 06:21:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 06:21:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886348 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886348 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16886348 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-07 06:21:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-07 06:21:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 06:21:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:08 proc_lsst.multi:170 [INFO] [multi] job_id 16886349 proc_lsst.multi INFO: [multi] job_id 16886349 2024-03-07 06:21:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 06:21:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886349 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886349 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16886349 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-07 06:21:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-07 06:21:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 06:21:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:09 proc_lsst.multi:170 [INFO] [multi] job_id 16886350 proc_lsst.multi INFO: [multi] job_id 16886350 2024-03-07 06:21:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 06:21:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886350 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886350 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16886350 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-07 06:21:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:09 proc_lsst.multi:170 [INFO] [multi] job_id 16886351 proc_lsst.multi INFO: [multi] job_id 16886351 2024-03-07 06:21:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 06:21:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886351 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886351 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16886351 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-07 06:21:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:09 proc_lsst.multi:170 [INFO] [multi] job_id 16886352 proc_lsst.multi INFO: [multi] job_id 16886352 2024-03-07 06:21:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 06:21:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886352 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886352 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16886352 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-07 06:21:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:09 proc_lsst.multi:170 [INFO] [multi] job_id 16886353 proc_lsst.multi INFO: [multi] job_id 16886353 2024-03-07 06:21:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 06:21:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886353 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886353 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16886353 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-07 06:21:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:09 proc_lsst.multi:170 [INFO] [multi] job_id 16886354 proc_lsst.multi INFO: [multi] job_id 16886354 2024-03-07 06:21:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 06:21:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886354 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886354 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16886354 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-07 06:21:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:09 proc_lsst.multi:170 [INFO] [multi] job_id 16886355 proc_lsst.multi INFO: [multi] job_id 16886355 2024-03-07 06:21:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 06:21:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886355 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886355 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16886355 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-07 06:21:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:09 proc_lsst.multi:170 [INFO] [multi] job_id 16886356 proc_lsst.multi INFO: [multi] job_id 16886356 2024-03-07 06:21:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 06:21:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886356 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886356 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16886356 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-07 06:21:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:09 proc_lsst.multi:170 [INFO] [multi] job_id 16886357 proc_lsst.multi INFO: [multi] job_id 16886357 2024-03-07 06:21:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 06:21:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886357 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886357 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16886357 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-07 06:21:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 06:21:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:10 proc_lsst.multi:170 [INFO] [multi] job_id 16886358 proc_lsst.multi INFO: [multi] job_id 16886358 2024-03-07 06:21:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 06:21:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886358 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886358 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16886358 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-07 06:21:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-07 06:21:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 06:21:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 06:21:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 06:21:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 06:21:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 06:21:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,169.254.95.120,n3009,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54443 --result_port=54406 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 06:21:10 proc_lsst.multi:170 [INFO] [multi] job_id 16886359 proc_lsst.multi INFO: [multi] job_id 16886359 2024-03-07 06:21:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 06:21:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 06:21:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16886359 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16886359 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16886359 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 124 parsl.dataflow.strategy DEBUG: Executor multi has 124 active tasks, 1/30 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 032e559ff9a0fa76680d3a69d6929a35 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/6c5cad59-d235-4b9a-81ec-d6d0f0468d11_isrForCrosstalkSources_1124443_21.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/6c5cad59-d235-4b9a-81ec-d6d0f0468d11_isrForCrosstalkSources_1124443_21.stderr parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39de1270fc12a2eff70050de43cb57cd with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/f50e8f63-e152-4115-83c6-040833451b18_isrForCrosstalkSources_1124443_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/f50e8f63-e152-4115-83c6-040833451b18_isrForCrosstalkSources_1124443_48.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 122 parsl.dataflow.strategy DEBUG: Executor multi has 122 active tasks, 1/30 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 122 parsl.dataflow.strategy DEBUG: Executor multi has 122 active tasks, 1/30 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3242ab9b2fe7761e1acbfd3d2c8ffa4c with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/5eb11b83-2f14-4beb-9380-dd9e765b1493_isrForCrosstalkSources_1124443_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/5eb11b83-2f14-4beb-9380-dd9e765b1493_isrForCrosstalkSources_1124443_4.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d364a127bf687bf13fbb0f1813db42d with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/f35ac62f-b778-4cd3-87fb-2e4440c8f32a_isrForCrosstalkSources_1124445_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/f35ac62f-b778-4cd3-87fb-2e4440c8f32a_isrForCrosstalkSources_1124445_37.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 120 parsl.dataflow.strategy DEBUG: Executor multi has 120 active tasks, 1/30 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17e1add9c3a6f4093e674ebdc97cd987 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/187eae1f-ef3d-491a-8fcf-0cd225c1f931_isrForCrosstalkSources_1124443_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/187eae1f-ef3d-491a-8fcf-0cd225c1f931_isrForCrosstalkSources_1124443_34.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38a9986153abc1b2e94baf44c0c31d31 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/f89fdf8a-4fc4-4dc1-aab9-2555bfff5816_isrForCrosstalkSources_1124445_50.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/f89fdf8a-4fc4-4dc1-aab9-2555bfff5816_isrForCrosstalkSources_1124445_50.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 118 parsl.dataflow.strategy DEBUG: Executor multi has 118 active tasks, 1/30 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 2024-03-07 06:21:35 proc_lsst.multi:146 [INFO] found job 64667 in provider local proc_lsst.multi INFO: found job 64667 in provider local 2024-03-07 06:21:35 proc_lsst.multi:146 [INFO] found job 16886330 in provider astro proc_lsst.multi INFO: found job 16886330 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3085c9d276bee28629fd7f5acccd3b97 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/98ac31c1-f5d2-4c52-9172-522e57c73bbc_isrForCrosstalkSources_1124445_27.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/98ac31c1-f5d2-4c52-9172-522e57c73bbc_isrForCrosstalkSources_1124445_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:36 proc_lsst.multi:146 [INFO] found job 16886331 in provider astro proc_lsst.multi INFO: found job 16886331 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:36 proc_lsst.multi:146 [INFO] found job 16886332 in provider astro proc_lsst.multi INFO: found job 16886332 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:36 proc_lsst.multi:146 [INFO] found job 16886333 in provider astro proc_lsst.multi INFO: found job 16886333 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e6743c72ef43ddca57b43b7581a3c23 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/4b0e774d-447b-4320-a39e-c11f2b69f361_isrForCrosstalkSources_1124445_8.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/4b0e774d-447b-4320-a39e-c11f2b69f361_isrForCrosstalkSources_1124445_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:37 proc_lsst.multi:146 [INFO] found job 16886334 in provider astro proc_lsst.multi INFO: found job 16886334 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:37 proc_lsst.multi:146 [INFO] found job 16886335 in provider astro proc_lsst.multi INFO: found job 16886335 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:37 proc_lsst.multi:146 [INFO] found job 16886336 in provider astro proc_lsst.multi INFO: found job 16886336 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:37 proc_lsst.multi:146 [INFO] found job 16886337 in provider astro proc_lsst.multi INFO: found job 16886337 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:38 proc_lsst.multi:146 [INFO] found job 16886338 in provider astro proc_lsst.multi INFO: found job 16886338 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:38 proc_lsst.multi:146 [INFO] found job 16886339 in provider astro proc_lsst.multi INFO: found job 16886339 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:39 proc_lsst.multi:146 [INFO] found job 16886340 in provider astro proc_lsst.multi INFO: found job 16886340 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:39 proc_lsst.multi:146 [INFO] found job 16886341 in provider astro proc_lsst.multi INFO: found job 16886341 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:39 proc_lsst.multi:146 [INFO] found job 16886342 in provider astro proc_lsst.multi INFO: found job 16886342 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:39 proc_lsst.multi:146 [INFO] found job 16886343 in provider astro proc_lsst.multi INFO: found job 16886343 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:40 proc_lsst.multi:146 [INFO] found job 16886344 in provider astro proc_lsst.multi INFO: found job 16886344 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:40 proc_lsst.multi:146 [INFO] found job 16886345 in provider astro proc_lsst.multi INFO: found job 16886345 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:40 proc_lsst.multi:146 [INFO] found job 16886346 in provider astro proc_lsst.multi INFO: found job 16886346 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:41 proc_lsst.multi:146 [INFO] found job 16886347 in provider astro proc_lsst.multi INFO: found job 16886347 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:41 proc_lsst.multi:146 [INFO] found job 16886348 in provider astro proc_lsst.multi INFO: found job 16886348 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:41 proc_lsst.multi:146 [INFO] found job 16886349 in provider astro proc_lsst.multi INFO: found job 16886349 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:41 proc_lsst.multi:146 [INFO] found job 16886350 in provider astro proc_lsst.multi INFO: found job 16886350 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:41 proc_lsst.multi:146 [INFO] found job 16886351 in provider astro proc_lsst.multi INFO: found job 16886351 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:42 proc_lsst.multi:146 [INFO] found job 16886352 in provider astro proc_lsst.multi INFO: found job 16886352 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:42 proc_lsst.multi:146 [INFO] found job 16886353 in provider astro proc_lsst.multi INFO: found job 16886353 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:42 proc_lsst.multi:146 [INFO] found job 16886354 in provider astro proc_lsst.multi INFO: found job 16886354 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:43 proc_lsst.multi:146 [INFO] found job 16886355 in provider astro proc_lsst.multi INFO: found job 16886355 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:43 proc_lsst.multi:146 [INFO] found job 16886356 in provider astro proc_lsst.multi INFO: found job 16886356 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:44 proc_lsst.multi:146 [INFO] found job 16886357 in provider astro proc_lsst.multi INFO: found job 16886357 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:44 proc_lsst.multi:146 [INFO] found job 16886358 in provider astro proc_lsst.multi INFO: found job 16886358 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:21:44 proc_lsst.multi:146 [INFO] found job 16886359 in provider astro proc_lsst.multi INFO: found job 16886359 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 116 parsl.dataflow.strategy DEBUG: Executor multi has 116 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 116 parsl.dataflow.strategy DEBUG: Executor multi has 116 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3809dd18c7b67c84c2c8230d91f65a0d with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/6d9a116f-3dc2-4aa8-871d-955820682d1a_isrForCrosstalkSources_1124443_19.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/6d9a116f-3dc2-4aa8-871d-955820682d1a_isrForCrosstalkSources_1124443_19.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff49d46491fa1ad086831d467a865ea0 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/be32d4cf-77a6-4d0d-9ade-aef35fd1d7e0_isrForCrosstalkSources_1124443_31.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/be32d4cf-77a6-4d0d-9ade-aef35fd1d7e0_isrForCrosstalkSources_1124443_31.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 114 parsl.dataflow.strategy DEBUG: Executor multi has 114 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6b47f4c63a56027920a03007ada431a with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/908519c7-5afc-4ea9-95ea-13d96820f911_isrForCrosstalkSources_1124445_15.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/908519c7-5afc-4ea9-95ea-13d96820f911_isrForCrosstalkSources_1124445_15.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31b060146f6cc2c2f05c6cb0be15a90e with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/a66797ff-98f3-469f-af3a-f138162ae55d_isrForCrosstalkSources_1124445_19.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/a66797ff-98f3-469f-af3a-f138162ae55d_isrForCrosstalkSources_1124445_19.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 112 parsl.dataflow.strategy DEBUG: Executor multi has 112 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcdecb3d48036982f04c746e7dd4862a with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/631186c9-70ef-4f02-93f2-b7962edb5be1_isrForCrosstalkSources_1124443_22.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/631186c9-70ef-4f02-93f2-b7962edb5be1_isrForCrosstalkSources_1124443_22.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89f733aefdd855770197a17af57f9991 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/6ecdce67-5018-4dab-9b3d-4fdd52b25ac1_isrForCrosstalkSources_1124445_43.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/6ecdce67-5018-4dab-9b3d-4fdd52b25ac1_isrForCrosstalkSources_1124445_43.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82d3f2ac4622487a009646e7833181c8 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/ad4cc67c-4a55-4e95-bdc7-0f14fff8ed39_isrForCrosstalkSources_1124445_5.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/ad4cc67c-4a55-4e95-bdc7-0f14fff8ed39_isrForCrosstalkSources_1124445_5.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dd8016441969cf25002cdedc03c9fe0 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/42908c6e-4f48-47f7-8fb6-933ca0c0fe40_isrForCrosstalkSources_1124443_43.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/42908c6e-4f48-47f7-8fb6-933ca0c0fe40_isrForCrosstalkSources_1124443_43.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55c5a679b799346a5e0301bd21a69110 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/be1928fa-a0e6-45f5-b8fa-8c22baf6a252_isrForCrosstalkSources_1124443_35.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/be1928fa-a0e6-45f5-b8fa-8c22baf6a252_isrForCrosstalkSources_1124443_35.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70f26a2580b4ac18ebb1331149e805cc with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/baffd39e-71ba-4c5f-9d0b-79f226e026e4_isrForCrosstalkSources_1124445_17.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/baffd39e-71ba-4c5f-9d0b-79f226e026e4_isrForCrosstalkSources_1124445_17.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa3f9cc375db67336bb157dbbd90f29d with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/972a98e9-7434-46a7-97a4-6f9c3eeda4d0_isrForCrosstalkSources_1124445_6.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/972a98e9-7434-46a7-97a4-6f9c3eeda4d0_isrForCrosstalkSources_1124445_6.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cde0f1573866d31f7d9ed12af77bfe30 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/caa52a88-13b1-4b52-8d32-59e66fbf2f70_isrForCrosstalkSources_1124445_41.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/caa52a88-13b1-4b52-8d32-59e66fbf2f70_isrForCrosstalkSources_1124445_41.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ff478d0cbe5240340a8f7385a05a330 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/9c5534dd-a4ba-4017-87c8-7947fcce6b3d_isrForCrosstalkSources_1124445_24.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/9c5534dd-a4ba-4017-87c8-7947fcce6b3d_isrForCrosstalkSources_1124445_24.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3493c988a7a6c32ec7eecd36667525f9 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/61630073-3ab3-43b0-81ea-5e468a2d3ed6_isrForCrosstalkSources_1124443_14.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/61630073-3ab3-43b0-81ea-5e468a2d3ed6_isrForCrosstalkSources_1124443_14.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 660ef2066d6f4877e3f0e0feb391908d with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/39d54a05-e190-4b10-98ba-8411cb4f81b6_isrForCrosstalkSources_1124443_13.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/39d54a05-e190-4b10-98ba-8411cb4f81b6_isrForCrosstalkSources_1124443_13.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 825c9919b4b71a3cb48a94af6d784ae2 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/b0fa308c-561b-4720-9efa-9df51e57db3b_isrForCrosstalkSources_1124445_4.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/b0fa308c-561b-4720-9efa-9df51e57db3b_isrForCrosstalkSources_1124445_4.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77b074f2b65cb0a2e82c8a4a8a3d9789 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/d2f56516-9d49-4c12-b53c-3658c833ab9c_isrForCrosstalkSources_1124445_32.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/d2f56516-9d49-4c12-b53c-3658c833ab9c_isrForCrosstalkSources_1124445_32.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55ae34fe13786356c6260c680fa45d76 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/f8796eae-3f92-4a13-8f76-ee80b70a1f8c_isrForCrosstalkSources_1124445_46.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/f8796eae-3f92-4a13-8f76-ee80b70a1f8c_isrForCrosstalkSources_1124445_46.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fcb35596bf2960ed6b82f302feed1e7 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/facfcad1-7a6a-4d03-9ea3-7c1330cec9c9_isrForCrosstalkSources_1124443_33.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/facfcad1-7a6a-4d03-9ea3-7c1330cec9c9_isrForCrosstalkSources_1124443_33.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 319bb3d106ac123219fdd4f79fc71961 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/d4601d22-fc42-4ff5-8ebd-3a99ad740440_isrForCrosstalkSources_1124445_39.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/d4601d22-fc42-4ff5-8ebd-3a99ad740440_isrForCrosstalkSources_1124445_39.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd3fc5e338269a8867b91110e475479c with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b1e2385c-343e-441f-92e2-34fceb67d14b_isrForCrosstalkSources_1124443_25.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b1e2385c-343e-441f-92e2-34fceb67d14b_isrForCrosstalkSources_1124443_25.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e432cfa9d2e4b7a46150bd4ec4e389d6 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/151bb07d-d456-4ddd-89cc-6285b2f3993b_isrForCrosstalkSources_1124445_51.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/151bb07d-d456-4ddd-89cc-6285b2f3993b_isrForCrosstalkSources_1124445_51.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1abd8e5788849ce6c0298fdabcad95b8 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5603b68f-e7d0-43c7-b1c1-ad32d6168dbc_isrForCrosstalkSources_1124445_18.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5603b68f-e7d0-43c7-b1c1-ad32d6168dbc_isrForCrosstalkSources_1124445_18.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67bd8fb0c03383a5b0e1879749ccd2bb with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/22f9d3a8-57e3-4be8-ae29-828bc999ccda_isrForCrosstalkSources_1124443_11.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/22f9d3a8-57e3-4be8-ae29-828bc999ccda_isrForCrosstalkSources_1124443_11.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1b3f1f437f1f7dffb6e86a13d429ff4 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ef81bc20-8199-4bdb-9ddf-8329a332d10e_isrForCrosstalkSources_1124443_6.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ef81bc20-8199-4bdb-9ddf-8329a332d10e_isrForCrosstalkSources_1124443_6.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f53b5756e578c7410d1e90424711e813 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/7824f045-ab10-4cc4-b8e0-628c833c97d9_isrForCrosstalkSources_1124445_13.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/7824f045-ab10-4cc4-b8e0-628c833c97d9_isrForCrosstalkSources_1124445_13.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6d58839fa4e00af5d4873cc43c357dc with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/0fcbc47c-0d21-4af9-b3d4-60e669bb3e4a_isrForCrosstalkSources_1124443_3.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/0fcbc47c-0d21-4af9-b3d4-60e669bb3e4a_isrForCrosstalkSources_1124443_3.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 578d6d5c32fecebde8b659ab2c472c09 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/ec7e34fe-1afe-459f-b8ee-1f2dc49a78e5_isrForCrosstalkSources_1124445_58.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/ec7e34fe-1afe-459f-b8ee-1f2dc49a78e5_isrForCrosstalkSources_1124445_58.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b404a4d2946e96886e2fe2fedd5d968 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/a88ed264-1c32-4eac-a274-a79b8a7d84f5_isrForCrosstalkSources_1124443_36.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/a88ed264-1c32-4eac-a274-a79b8a7d84f5_isrForCrosstalkSources_1124443_36.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f50fff995e994c05ddaca3b3101a4196 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/573dba41-7a4f-4fa1-b91a-d62154d3a2f4_isrForCrosstalkSources_1124445_35.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/573dba41-7a4f-4fa1-b91a-d62154d3a2f4_isrForCrosstalkSources_1124445_35.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 86 parsl.dataflow.strategy DEBUG: Executor multi has 86 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef66adc19e165caa1341e336714244f4 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/8922fb0f-d524-48d4-a4c7-ac1e7fbb2778_isrForCrosstalkSources_1124443_44.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/8922fb0f-d524-48d4-a4c7-ac1e7fbb2778_isrForCrosstalkSources_1124443_44.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e45326dbb7accae9d8b49ddd08e990eb with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ef07ffc4-96e6-4603-88e5-860641af4e08_isrForCrosstalkSources_1124443_42.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ef07ffc4-96e6-4603-88e5-860641af4e08_isrForCrosstalkSources_1124443_42.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2cb666202a2c188f043381467c710c8 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/34239b95-ccdc-4165-8e63-7db478591913_isrForCrosstalkSources_1124443_49.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/34239b95-ccdc-4165-8e63-7db478591913_isrForCrosstalkSources_1124443_49.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0612fc94e7eaa158f003dcd6bc00930a with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/97d3b131-8eaf-4d14-ba3c-6ef9e3f61d60_isrForCrosstalkSources_1124443_54.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/97d3b131-8eaf-4d14-ba3c-6ef9e3f61d60_isrForCrosstalkSources_1124443_54.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c38d589dd5e7d9fe49299cf17e8a395 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/fb01d323-ba6d-4eba-89cc-f6340d4ca044_isrForCrosstalkSources_1124445_57.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/fb01d323-ba6d-4eba-89cc-f6340d4ca044_isrForCrosstalkSources_1124445_57.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d22d66e3fea606a0b8310395db3e543 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/6d52ec1f-f461-4bd9-bf15-e8504b0cbf9b_isrForCrosstalkSources_1124445_42.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/6d52ec1f-f461-4bd9-bf15-e8504b0cbf9b_isrForCrosstalkSources_1124445_42.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 80 parsl.dataflow.strategy DEBUG: Executor multi has 80 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c0720dfec9ea3f66fd3ec0ece39a1e8 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/33f177f4-0a1e-4b5c-998a-6985ff55e7eb_isrForCrosstalkSources_1124445_12.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/33f177f4-0a1e-4b5c-998a-6985ff55e7eb_isrForCrosstalkSources_1124445_12.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ec9318a308e95b7519d019f3ac2dd76 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/c8fb8957-71f1-4546-84a4-171ae0019e9e_isrForCrosstalkSources_1124445_14.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/c8fb8957-71f1-4546-84a4-171ae0019e9e_isrForCrosstalkSources_1124445_14.stderr 2024-03-07 06:22:09 proc_lsst.multi:146 [INFO] found job 64667 in provider local proc_lsst.multi INFO: found job 64667 in provider local 2024-03-07 06:22:10 proc_lsst.multi:146 [INFO] found job 16886330 in provider astro proc_lsst.multi INFO: found job 16886330 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be9bde870b5d03414b337bee95dbe2e6 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ba0c7231-c869-495b-94fd-dbfaf193857e_isrForCrosstalkSources_1124443_32.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ba0c7231-c869-495b-94fd-dbfaf193857e_isrForCrosstalkSources_1124443_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:10 proc_lsst.multi:146 [INFO] found job 16886331 in provider astro proc_lsst.multi INFO: found job 16886331 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bb331a77a91c1af8e698747a05500df with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/2eedaa37-2225-4341-b4fa-9be50ca5f86e_isrForCrosstalkSources_1124445_20.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/2eedaa37-2225-4341-b4fa-9be50ca5f86e_isrForCrosstalkSources_1124445_20.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cd8bdc2ed741a1efe6ce17f654f4e59 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5f0e19af-0add-440a-a4c0-47b526b9c62f_isrForCrosstalkSources_1124445_23.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5f0e19af-0add-440a-a4c0-47b526b9c62f_isrForCrosstalkSources_1124445_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:10 proc_lsst.multi:146 [INFO] found job 16886332 in provider astro proc_lsst.multi INFO: found job 16886332 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61a5b8a0f4d403f6a1c8757b5d987ecc with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/5be3a009-cfce-4673-9a8d-aee1fc5730dc_isrForCrosstalkSources_1124443_5.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/5be3a009-cfce-4673-9a8d-aee1fc5730dc_isrForCrosstalkSources_1124443_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:11 proc_lsst.multi:146 [INFO] found job 16886333 in provider astro proc_lsst.multi INFO: found job 16886333 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:11 proc_lsst.multi:146 [INFO] found job 16886334 in provider astro proc_lsst.multi INFO: found job 16886334 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e1fa682b6d11328fb8982c991239d65 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/bcfaeb96-ffbd-45b9-871e-21ebf2c4f28b_isrForCrosstalkSources_1124445_9.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/bcfaeb96-ffbd-45b9-871e-21ebf2c4f28b_isrForCrosstalkSources_1124445_9.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 473e50d4af6b3612da442a1e8af7070d with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/258910a1-85bf-409e-88e4-44594455e3de_isrForCrosstalkSources_1124443_38.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/258910a1-85bf-409e-88e4-44594455e3de_isrForCrosstalkSources_1124443_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:11 proc_lsst.multi:146 [INFO] found job 16886335 in provider astro proc_lsst.multi INFO: found job 16886335 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 285982ae19e046ca686d65ec22ab735f with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/8d1dbbd4-7694-4e0d-acbf-9e058f3d8ffe_isrForCrosstalkSources_1124443_28.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/8d1dbbd4-7694-4e0d-acbf-9e058f3d8ffe_isrForCrosstalkSources_1124443_28.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 945073ded0f5c911ceb16f2589fac59a with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/7340798f-805a-4821-b640-42b250b1cfa2_isrForCrosstalkSources_1124443_26.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/7340798f-805a-4821-b640-42b250b1cfa2_isrForCrosstalkSources_1124443_26.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b9fe3babd6085d297a6d55a71af9cec with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/f7e72421-82b5-4b15-95bb-5b82a6177216_isrForCrosstalkSources_1124443_46.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/f7e72421-82b5-4b15-95bb-5b82a6177216_isrForCrosstalkSources_1124443_46.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a1677c94731abf2e176c58a857eb2dd with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/c5d176dc-a5ea-440b-ada0-d6fa638cde42_isrForCrosstalkSources_1124443_27.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/c5d176dc-a5ea-440b-ada0-d6fa638cde42_isrForCrosstalkSources_1124443_27.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 265210e01fa585c979e8d8ec61d789e0 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ddbc1786-0efd-43ae-a3ad-36c95589cf2b_isrForCrosstalkSources_1124443_16.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/ddbc1786-0efd-43ae-a3ad-36c95589cf2b_isrForCrosstalkSources_1124443_16.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1569076282dd4a0bcd61ec8cb519ab19 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/83c327b0-41d7-4c63-8bb5-a7e0147c3372_isrForCrosstalkSources_1124445_21.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/83c327b0-41d7-4c63-8bb5-a7e0147c3372_isrForCrosstalkSources_1124445_21.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30d4e24b05b9d3025eae2b6d66205e99 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b0512b34-65e4-45da-9f09-6a7ea98e53e9_isrForCrosstalkSources_1124443_56.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b0512b34-65e4-45da-9f09-6a7ea98e53e9_isrForCrosstalkSources_1124443_56.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 661308e64d5f787a384f3cd533947d88 with result from task 63 parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:11 proc_lsst.multi:146 [INFO] found job 16886336 in provider astro proc_lsst.multi INFO: found job 16886336 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/e938f921-4354-4f21-a793-d8f9f3911f8a_isrForCrosstalkSources_1124443_8.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/e938f921-4354-4f21-a793-d8f9f3911f8a_isrForCrosstalkSources_1124443_8.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d2ef17cf30b2b24ebc2b71f8dc4b54f with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/3af8ad10-87e7-4241-9551-bbd327be06ab_isrForCrosstalkSources_1124443_10.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/3af8ad10-87e7-4241-9551-bbd327be06ab_isrForCrosstalkSources_1124443_10.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfa997494e4c1b87a9a433e8878a3e22 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/e55c2f22-20a3-4fae-ba52-b414d993d339_isrForCrosstalkSources_1124443_18.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/e55c2f22-20a3-4fae-ba52-b414d993d339_isrForCrosstalkSources_1124443_18.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9508e6cffaf6956868c847965a9ef8d with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/de336afa-ca4b-47e6-b8fc-2c2580a4fd5e_isrForCrosstalkSources_1124445_25.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/de336afa-ca4b-47e6-b8fc-2c2580a4fd5e_isrForCrosstalkSources_1124445_25.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8322b581821d18bedb51624e826c1cdd with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/19fae75c-5e44-4453-b8ec-caeb3ff8822e_isrForCrosstalkSources_1124445_33.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/19fae75c-5e44-4453-b8ec-caeb3ff8822e_isrForCrosstalkSources_1124445_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:11 proc_lsst.multi:146 [INFO] found job 16886337 in provider astro proc_lsst.multi INFO: found job 16886337 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0f3f5967ff3994256bee7ec1039da28 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/c3991f15-8e1a-4f10-9c3b-bedfa4a81de4_isrForCrosstalkSources_1124443_53.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/c3991f15-8e1a-4f10-9c3b-bedfa4a81de4_isrForCrosstalkSources_1124443_53.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c73045d1d25c822c8b16766569dedc03 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/7464598a-88a8-40b2-afe5-34d129242b2f_isrForCrosstalkSources_1124445_11.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/7464598a-88a8-40b2-afe5-34d129242b2f_isrForCrosstalkSources_1124445_11.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54c5d1d22b88a43820176b3bb0c7fc41 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/54ffa00b-b643-4177-8340-06eaeb34e6b5_isrForCrosstalkSources_1124445_44.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/54ffa00b-b643-4177-8340-06eaeb34e6b5_isrForCrosstalkSources_1124445_44.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4369021963ca0479c22d867c64b2fe9d with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/65cc0942-9990-4205-9d14-3c01f7226abb_isrForCrosstalkSources_1124443_30.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/65cc0942-9990-4205-9d14-3c01f7226abb_isrForCrosstalkSources_1124443_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:12 proc_lsst.multi:146 [INFO] found job 16886338 in provider astro proc_lsst.multi INFO: found job 16886338 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 675d81a164fda6f4bcbac707e4f70885 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b07d4dc4-72d1-4c27-93cb-b35426abe73f_isrForCrosstalkSources_1124443_41.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b07d4dc4-72d1-4c27-93cb-b35426abe73f_isrForCrosstalkSources_1124443_41.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 990a7dec42e63c804a551216366d770c with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/127d89b8-3c56-4121-8b12-3aa325d9cfb6_isrForCrosstalkSources_1124443_29.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/127d89b8-3c56-4121-8b12-3aa325d9cfb6_isrForCrosstalkSources_1124443_29.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2b33e5f1550c4487b446326986f4b36 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/9da4cdc1-5502-4103-b045-cc480022be43_isrForCrosstalkSources_1124445_48.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/9da4cdc1-5502-4103-b045-cc480022be43_isrForCrosstalkSources_1124445_48.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e88130dc939e79742d634a2685a057f with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/1e41c3b9-cd23-448b-a9d0-6cceb5e9466d_isrForCrosstalkSources_1124443_50.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/1e41c3b9-cd23-448b-a9d0-6cceb5e9466d_isrForCrosstalkSources_1124443_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:12 proc_lsst.multi:146 [INFO] found job 16886339 in provider astro proc_lsst.multi INFO: found job 16886339 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:12 proc_lsst.multi:146 [INFO] found job 16886340 in provider astro proc_lsst.multi INFO: found job 16886340 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 249fb32acf1dc4028f821fec4265b6df with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/79e3dbea-2d8c-47b0-9982-54a426e6cd47_isrForCrosstalkSources_1124445_31.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/79e3dbea-2d8c-47b0-9982-54a426e6cd47_isrForCrosstalkSources_1124445_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:12 proc_lsst.multi:146 [INFO] found job 16886341 in provider astro proc_lsst.multi INFO: found job 16886341 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:12 proc_lsst.multi:146 [INFO] found job 16886342 in provider astro proc_lsst.multi INFO: found job 16886342 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67fbfaa6890105bd031245b2af76b0a4 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b4e10b32-4a66-4c76-a9e2-74d975bbe6df_isrForCrosstalkSources_1124443_59.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b4e10b32-4a66-4c76-a9e2-74d975bbe6df_isrForCrosstalkSources_1124443_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:12 proc_lsst.multi:146 [INFO] found job 16886343 in provider astro proc_lsst.multi INFO: found job 16886343 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b7282d261348ba60bd579bff707eab6 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/42b994f4-0af5-4eeb-857f-15517d80593f_isrForCrosstalkSources_1124443_15.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/42b994f4-0af5-4eeb-857f-15517d80593f_isrForCrosstalkSources_1124443_15.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34800269173fbcd3f7e658f58cad8e3e with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/b537f6a3-1766-4851-9c77-0a1e39d7a9e4_isrForCrosstalkSources_1124445_28.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/b537f6a3-1766-4851-9c77-0a1e39d7a9e4_isrForCrosstalkSources_1124445_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:13 proc_lsst.multi:146 [INFO] found job 16886344 in provider astro proc_lsst.multi INFO: found job 16886344 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:13 proc_lsst.multi:146 [INFO] found job 16886345 in provider astro proc_lsst.multi INFO: found job 16886345 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:14 proc_lsst.multi:146 [INFO] found job 16886346 in provider astro proc_lsst.multi INFO: found job 16886346 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:14 proc_lsst.multi:146 [INFO] found job 16886347 in provider astro proc_lsst.multi INFO: found job 16886347 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:14 proc_lsst.multi:146 [INFO] found job 16886348 in provider astro proc_lsst.multi INFO: found job 16886348 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:15 proc_lsst.multi:146 [INFO] found job 16886349 in provider astro proc_lsst.multi INFO: found job 16886349 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:15 proc_lsst.multi:146 [INFO] found job 16886350 in provider astro proc_lsst.multi INFO: found job 16886350 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:15 proc_lsst.multi:146 [INFO] found job 16886351 in provider astro proc_lsst.multi INFO: found job 16886351 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9822b701501f640d20db9ec4714fc167 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/0cac22f2-d851-4fec-aa7a-93d67bc8499d_isrForCrosstalkSources_1124443_58.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/0cac22f2-d851-4fec-aa7a-93d67bc8499d_isrForCrosstalkSources_1124443_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:15 proc_lsst.multi:146 [INFO] found job 16886352 in provider astro proc_lsst.multi INFO: found job 16886352 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:16 proc_lsst.multi:146 [INFO] found job 16886353 in provider astro proc_lsst.multi INFO: found job 16886353 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:16 proc_lsst.multi:146 [INFO] found job 16886354 in provider astro proc_lsst.multi INFO: found job 16886354 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:16 proc_lsst.multi:146 [INFO] found job 16886355 in provider astro proc_lsst.multi INFO: found job 16886355 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:16 proc_lsst.multi:146 [INFO] found job 16886356 in provider astro proc_lsst.multi INFO: found job 16886356 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:17 proc_lsst.multi:146 [INFO] found job 16886357 in provider astro proc_lsst.multi INFO: found job 16886357 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3820893921c3b5f5f7075f48fcf92266 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5891b97f-d6e0-4770-ba08-bc0353f7c171_isrForCrosstalkSources_1124445_62.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5891b97f-d6e0-4770-ba08-bc0353f7c171_isrForCrosstalkSources_1124445_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:17 proc_lsst.multi:146 [INFO] found job 16886358 in provider astro proc_lsst.multi INFO: found job 16886358 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING 2024-03-07 06:22:17 proc_lsst.multi:146 [INFO] found job 16886359 in provider astro proc_lsst.multi INFO: found job 16886359 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16886330,16886331,16886332,16886333,16886334,16886335,16886336,16886337,16886338,16886339,16886340,16886341,16886342,16886343,16886344,16886345,16886346,16886347,16886348,16886349,16886350,16886351,16886352,16886353,16886354,16886355,16886356,16886357,16886358,16886359' parsl.providers.slurm.slurm DEBUG: squeue returned 16886355 R 16886356 R 16886357 R 16886358 R 16886359 R 16886345 R 16886346 R 16886347 R 16886348 R 16886349 R 16886350 R 16886351 R 16886352 R 16886353 R 16886354 R 16886335 R 16886336 R 16886337 R 16886338 R 16886339 R 16886340 R 16886341 R 16886342 R 16886343 R 16886344 R 16886330 R 16886331 R 16886332 R 16886333 R 16886334 R parsl.providers.slurm.slurm DEBUG: Updating job 16886355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16886334 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 46 parsl.dataflow.strategy DEBUG: Executor multi has 46 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 46 parsl.dataflow.strategy DEBUG: Executor multi has 46 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99c6aea370d421344713df2b6c4ad1d2 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/defa20ce-9562-42bb-b54f-d1c4738a1108_isrForCrosstalkSources_1124445_7.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/defa20ce-9562-42bb-b54f-d1c4738a1108_isrForCrosstalkSources_1124445_7.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b8914fd6756998cdb1ca7aa291baf65 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/1e69ed03-ca05-4259-8542-ba7752339e53_isrForCrosstalkSources_1124443_1.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/1e69ed03-ca05-4259-8542-ba7752339e53_isrForCrosstalkSources_1124443_1.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb7d5f7c19df1956224f0f40727d2e5b with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/8fabb69d-da6c-4623-ba02-4e628d4e0fde_isrForCrosstalkSources_1124443_37.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/8fabb69d-da6c-4623-ba02-4e628d4e0fde_isrForCrosstalkSources_1124443_37.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef2c87aba88e61dc3fba77a8a46bb464 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/fce36c87-ecf5-43e7-999a-b0ab74d881d1_isrForCrosstalkSources_1124443_55.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/fce36c87-ecf5-43e7-999a-b0ab74d881d1_isrForCrosstalkSources_1124443_55.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e59f403420aed55a0e37d3c71209df1c with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/56228c9a-4343-4aca-86ec-04ca413b18a4_isrForCrosstalkSources_1124445_52.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/56228c9a-4343-4aca-86ec-04ca413b18a4_isrForCrosstalkSources_1124445_52.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa3390138ca251fe468f19809d2332b7 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/e08b47fe-aa7d-4bcb-9222-5d3e6c04ca86_isrForCrosstalkSources_1124445_53.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/e08b47fe-aa7d-4bcb-9222-5d3e6c04ca86_isrForCrosstalkSources_1124445_53.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2791f30002b4099b80c918990fcd6729 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/0201e746-5637-41a5-80b4-911bbd21e7e4_isrForCrosstalkSources_1124445_47.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/0201e746-5637-41a5-80b4-911bbd21e7e4_isrForCrosstalkSources_1124445_47.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d46d14912fac6a1eee912862007855a3 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/35c85dad-9eb0-43a0-95ac-e708b1602bfe_isrForCrosstalkSources_1124445_55.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/35c85dad-9eb0-43a0-95ac-e708b1602bfe_isrForCrosstalkSources_1124445_55.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c353958b5bd4c0bbfedffeb1eae67858 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/21eb5745-1b85-48d2-a998-3af617e820c2_isrForCrosstalkSources_1124445_61.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/21eb5745-1b85-48d2-a998-3af617e820c2_isrForCrosstalkSources_1124445_61.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed5f3f96dc2f4e65f1e77618b0009aad with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/01ca3867-73f3-4ce9-9119-ac23163e6de5_isrForCrosstalkSources_1124445_49.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/01ca3867-73f3-4ce9-9119-ac23163e6de5_isrForCrosstalkSources_1124445_49.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1574eeb41c50489e5644ecb499f555a with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/39e26e35-e7f3-4506-9bef-344c869aa704_isrForCrosstalkSources_1124443_23.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/39e26e35-e7f3-4506-9bef-344c869aa704_isrForCrosstalkSources_1124443_23.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bb54682f3f5f55ed5ba9888a1c090e2 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/6a63f590-a229-44e2-83d5-5d3425452447_isrForCrosstalkSources_1124445_45.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/6a63f590-a229-44e2-83d5-5d3425452447_isrForCrosstalkSources_1124445_45.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 34 parsl.dataflow.strategy DEBUG: Executor multi has 34 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 596b8729efc79f7dc8a5eb1efaaa3e76 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/a594a4d3-74dc-4c9d-a88d-5d01fb44aa51_isrForCrosstalkSources_1124443_24.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/a594a4d3-74dc-4c9d-a88d-5d01fb44aa51_isrForCrosstalkSources_1124443_24.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 822f0f3c8a2679a41e3955bc66428f14 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5fdea119-0ba5-45b5-9fd9-ff290e58fd6c_isrForCrosstalkSources_1124445_2.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5fdea119-0ba5-45b5-9fd9-ff290e58fd6c_isrForCrosstalkSources_1124445_2.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01e977d0fc70692874c298fc173a1a70 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/d9d6097b-5e1a-4899-b2a1-7b0077838223_isrForCrosstalkSources_1124445_3.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/d9d6097b-5e1a-4899-b2a1-7b0077838223_isrForCrosstalkSources_1124445_3.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f61c3d52484b87a2165f798958bc8ca6 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/43f7924b-f0d8-47a6-9c0f-ae38c1d71e2a_isrForCrosstalkSources_1124445_36.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/43f7924b-f0d8-47a6-9c0f-ae38c1d71e2a_isrForCrosstalkSources_1124445_36.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26c3e06c6e26f4d8c36098c710bcdeb2 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/04c05710-db12-4258-a385-a85dd822baa5_isrForCrosstalkSources_1124443_47.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/04c05710-db12-4258-a385-a85dd822baa5_isrForCrosstalkSources_1124443_47.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdeaea6f7acd3bf3c12ce0a1f56c48a9 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5c2b0760-14cf-4743-b7e9-9307304c54f6_isrForCrosstalkSources_1124445_16.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/5c2b0760-14cf-4743-b7e9-9307304c54f6_isrForCrosstalkSources_1124445_16.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4401b0a47944346d7daf0e86c6f0c3f8 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/4d25282a-5574-4a0f-8549-8bc32f8b02a5_isrForCrosstalkSources_1124445_34.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/4d25282a-5574-4a0f-8549-8bc32f8b02a5_isrForCrosstalkSources_1124445_34.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6820d096596a98b09b6dc0015be6920 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/e078871f-0631-43ff-a81a-dfc8022c24fe_isrForCrosstalkSources_1124445_26.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/e078871f-0631-43ff-a81a-dfc8022c24fe_isrForCrosstalkSources_1124445_26.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c54d90930f18be590bc772291fe18199 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/4a8051d1-dc7b-4e06-a04a-f22b3fcc4ef4_isrForCrosstalkSources_1124445_54.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/4a8051d1-dc7b-4e06-a04a-f22b3fcc4ef4_isrForCrosstalkSources_1124445_54.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16d03e22d94d000cd3d7bae983ef73d9 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/35359f61-7328-4583-9e8a-98eb2d35bf07_isrForCrosstalkSources_1124445_1.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/35359f61-7328-4583-9e8a-98eb2d35bf07_isrForCrosstalkSources_1124445_1.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a850d4ca1e58b12ddac3324e5a33f86d with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/251f6ed5-d86c-40e6-9066-e74ada49ec44_isrForCrosstalkSources_1124445_38.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/251f6ed5-d86c-40e6-9066-e74ada49ec44_isrForCrosstalkSources_1124445_38.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48193265e41a4c5fdb047bece403b451 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/809c427d-50a9-4bbe-8880-a0c1584c7583_isrForCrosstalkSources_1124443_60.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/809c427d-50a9-4bbe-8880-a0c1584c7583_isrForCrosstalkSources_1124443_60.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1fb2df058f0c7e3b270cfc6bf9fb112 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/958c3e36-a631-4b27-bb71-9cfd7c10c7f1_isrForCrosstalkSources_1124443_39.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/958c3e36-a631-4b27-bb71-9cfd7c10c7f1_isrForCrosstalkSources_1124443_39.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 785c0a4d7d885a2e045181d404a76f06 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/83c6c691-0da3-4417-b4e6-6ec755ce904c_isrForCrosstalkSources_1124445_30.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/83c6c691-0da3-4417-b4e6-6ec755ce904c_isrForCrosstalkSources_1124445_30.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 437f23a5d9fe2245aa84492eb0cb08b3 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/2fcb0bd4-128c-429c-853e-0ae88009a255_isrForCrosstalkSources_1124443_51.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/2fcb0bd4-128c-429c-853e-0ae88009a255_isrForCrosstalkSources_1124443_51.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5221df3c58467a5dd993ce1bd6987a1 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/2f25774d-6e04-4874-8ee3-142be0c77fe0_isrForCrosstalkSources_1124445_10.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/2f25774d-6e04-4874-8ee3-142be0c77fe0_isrForCrosstalkSources_1124445_10.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df18cc4dc8eeb0ec94a323fca042503a with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/5e141566-f4bb-4d7f-8c97-ea805522c579_isrForCrosstalkSources_1124443_2.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/5e141566-f4bb-4d7f-8c97-ea805522c579_isrForCrosstalkSources_1124443_2.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6c4b3cce5658f762dc06bbe7493e037 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/6071eb38-f66a-43de-a6f6-acfb4b6c7690_isrForCrosstalkSources_1124443_20.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/6071eb38-f66a-43de-a6f6-acfb4b6c7690_isrForCrosstalkSources_1124443_20.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ff053094b397b0afdc144ba8a3faa0f with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/9484492d-d370-4e7b-8b61-53e6ba13c394_isrForCrosstalkSources_1124443_57.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/9484492d-d370-4e7b-8b61-53e6ba13c394_isrForCrosstalkSources_1124443_57.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58e9c13bd35e72d21c2799032ecae734 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/53f58e41-5995-4d8b-b303-5faa664fd721_isrForCrosstalkSources_1124443_40.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/53f58e41-5995-4d8b-b303-5faa664fd721_isrForCrosstalkSources_1124443_40.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 14 parsl.dataflow.strategy DEBUG: Executor multi has 14 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e649f92dcc298c62a852a4859134078f with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/67ae6411-6ed1-48d9-a3c9-b83ea4153100_isrForCrosstalkSources_1124443_17.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/67ae6411-6ed1-48d9-a3c9-b83ea4153100_isrForCrosstalkSources_1124443_17.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aab418aa4a82c030cd9253f71a96eb97 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/4de241f0-5ec2-4485-82cb-350dea762798_isrForCrosstalkSources_1124443_45.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/4de241f0-5ec2-4485-82cb-350dea762798_isrForCrosstalkSources_1124443_45.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58c535eb7bdc903bb83571aa41288ea6 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/120912f4-1985-4e29-97a1-160c7151b4a7_isrForCrosstalkSources_1124443_52.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/120912f4-1985-4e29-97a1-160c7151b4a7_isrForCrosstalkSources_1124443_52.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60fe4d35129b5864a11c3d4ab74ec965 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/8e9ef3fa-c5b9-4286-bd90-56c26220e379_isrForCrosstalkSources_1124445_40.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/8e9ef3fa-c5b9-4286-bd90-56c26220e379_isrForCrosstalkSources_1124445_40.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bbb53a2f5c9e5debda405eee53fea8c with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/61b8365e-f923-44a1-aa0d-f8e4d7d5a08b_isrForCrosstalkSources_1124443_7.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/61b8365e-f923-44a1-aa0d-f8e4d7d5a08b_isrForCrosstalkSources_1124443_7.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e96324d167196abc715ef1152a05212 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/96aeef08-a3bc-452e-bf7a-e752dde87c42_isrForCrosstalkSources_1124443_9.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/96aeef08-a3bc-452e-bf7a-e752dde87c42_isrForCrosstalkSources_1124443_9.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 149dd07693ec036fc1c9fa0477d22554 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/8a3504fd-2248-4e80-a363-51c14ee7cf8c_isrForCrosstalkSources_1124445_22.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/8a3504fd-2248-4e80-a363-51c14ee7cf8c_isrForCrosstalkSources_1124445_22.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e2fac1692ea770aa9ccdf58f485a33e with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/81e78171-8e6b-4f7b-9e5a-eea19c9b39f1_isrForCrosstalkSources_1124443_12.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/81e78171-8e6b-4f7b-9e5a-eea19c9b39f1_isrForCrosstalkSources_1124443_12.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9d05ed8ea810c23671e42f17a57bbf3 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/773c8b3a-7ea9-465e-99f0-6830d81328bc_isrForCrosstalkSources_1124445_56.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/773c8b3a-7ea9-465e-99f0-6830d81328bc_isrForCrosstalkSources_1124445_56.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3295d346ad8feec1937f83d9e5264fdb with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/ed2de177-bf57-4730-90bf-013aab850023_isrForCrosstalkSources_1124445_29.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/ed2de177-bf57-4730-90bf-013aab850023_isrForCrosstalkSources_1124445_29.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f2ab5f52e7be15c6c6b56909004b62c with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/cb76d97a-d0e3-4d79-82db-71156c3cfba5_isrForCrosstalkSources_1124443_61.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/cb76d97a-d0e3-4d79-82db-71156c3cfba5_isrForCrosstalkSources_1124443_61.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c5253da3d5967ff3c21f4cd3bda39d0 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b50bb89c-ebdf-4c4e-ad7b-cd4ede3dff1c_isrForCrosstalkSources_1124443_62.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124443/b50bb89c-ebdf-4c4e-ad7b-cd4ede3dff1c_isrForCrosstalkSources_1124443_62.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0332368882178ffb8abd186fb4196754 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/49c69afd-775a-43a1-9b51-497b1f7e0cdb_isrForCrosstalkSources_1124445_59.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/49c69afd-775a-43a1-9b51-497b1f7e0cdb_isrForCrosstalkSources_1124445_59.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22838134267600 parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd03d0a84cf052552902a3e7d931af95 with result from task 111 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 124 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/65bda2d7-50d9-4ba2-bf63-53b7995b7880_isrForCrosstalkSources_1124445_60.stdout parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/submit/DEEP/20220822/B1m/science#step0/20240307T062031Z/logs/isrForCrosstalkSources/1124445/65bda2d7-50d9-4ba2-bf63-53b7995b7880_isrForCrosstalkSources_1124445_60.stderr parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=31, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 31 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3ceab9de3df3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3ceab9de3df3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c2ef9cf48333 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c2ef9cf48333 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 89151eb275c5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 89151eb275c5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cc6c23573947 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cc6c23573947 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: da1863c834a6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: da1863c834a6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4a8dbbddb0e4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4a8dbbddb0e4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 22aa78197ad6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 22aa78197ad6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7a9e6a542f44 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7a9e6a542f44 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 63f1a98afaa4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 63f1a98afaa4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7b36eed3e539 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7b36eed3e539 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c6305b85546c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c6305b85546c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 162b14bdce60 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 162b14bdce60 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 519a1bc57770 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 519a1bc57770 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d8934262fee1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d8934262fee1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8cccb33c2469 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8cccb33c2469 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a3416e4f6547 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a3416e4f6547 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 93a9d327887d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 93a9d327887d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f9e3cdda86d0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f9e3cdda86d0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 19369488238a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 19369488238a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e8eebe21e8b7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e8eebe21e8b7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6adcc0fcf23d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6adcc0fcf23d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7493b5203538 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7493b5203538 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4a3170dc74b0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4a3170dc74b0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 44ac20528f2c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 44ac20528f2c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf94d29712db parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf94d29712db parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4ab045f4c52a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4ab045f4c52a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9fc9ca0668ea parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9fc9ca0668ea parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 59952ee87e6f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 59952ee87e6f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e9cf9593487 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e9cf9593487 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0841a70b72a7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0841a70b72a7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1091b198e6b3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1091b198e6b3 2024-03-07 06:22:33 proc_lsst.multi:146 [INFO] found job 16886333 in provider astro proc_lsst.multi INFO: found job 16886333 in provider astro 2024-03-07 06:22:33 proc_lsst.multi:201 [INFO] cancelling 16886333 on provider astro proc_lsst.multi INFO: cancelling 16886333 on provider astro 2024-03-07 06:22:33 proc_lsst.multi:146 [INFO] found job 16886348 in provider astro proc_lsst.multi INFO: found job 16886348 in provider astro 2024-03-07 06:22:33 proc_lsst.multi:201 [INFO] cancelling 16886348 on provider astro proc_lsst.multi INFO: cancelling 16886348 on provider astro 2024-03-07 06:22:33 proc_lsst.multi:146 [INFO] found job 16886330 in provider astro proc_lsst.multi INFO: found job 16886330 in provider astro 2024-03-07 06:22:33 proc_lsst.multi:201 [INFO] cancelling 16886330 on provider astro proc_lsst.multi INFO: cancelling 16886330 on provider astro 2024-03-07 06:22:33 proc_lsst.multi:146 [INFO] found job 16886336 in provider astro proc_lsst.multi INFO: found job 16886336 in provider astro 2024-03-07 06:22:33 proc_lsst.multi:201 [INFO] cancelling 16886336 on provider astro proc_lsst.multi INFO: cancelling 16886336 on provider astro 2024-03-07 06:22:33 proc_lsst.multi:146 [INFO] found job 16886357 in provider astro proc_lsst.multi INFO: found job 16886357 in provider astro 2024-03-07 06:22:33 proc_lsst.multi:201 [INFO] cancelling 16886357 on provider astro proc_lsst.multi INFO: cancelling 16886357 on provider astro 2024-03-07 06:22:33 proc_lsst.multi:146 [INFO] found job 16886335 in provider astro proc_lsst.multi INFO: found job 16886335 in provider astro 2024-03-07 06:22:33 proc_lsst.multi:201 [INFO] cancelling 16886335 on provider astro proc_lsst.multi INFO: cancelling 16886335 on provider astro 2024-03-07 06:22:33 proc_lsst.multi:146 [INFO] found job 16886334 in provider astro proc_lsst.multi INFO: found job 16886334 in provider astro 2024-03-07 06:22:33 proc_lsst.multi:201 [INFO] cancelling 16886334 on provider astro proc_lsst.multi INFO: cancelling 16886334 on provider astro 2024-03-07 06:22:33 proc_lsst.multi:146 [INFO] found job 16886331 in provider astro proc_lsst.multi INFO: found job 16886331 in provider astro 2024-03-07 06:22:33 proc_lsst.multi:201 [INFO] cancelling 16886331 on provider astro proc_lsst.multi INFO: cancelling 16886331 on provider astro 2024-03-07 06:22:33 proc_lsst.multi:146 [INFO] found job 16886339 in provider astro proc_lsst.multi INFO: found job 16886339 in provider astro 2024-03-07 06:22:33 proc_lsst.multi:201 [INFO] cancelling 16886339 on provider astro proc_lsst.multi INFO: cancelling 16886339 on provider astro 2024-03-07 06:22:34 proc_lsst.multi:146 [INFO] found job 16886332 in provider astro proc_lsst.multi INFO: found job 16886332 in provider astro 2024-03-07 06:22:34 proc_lsst.multi:201 [INFO] cancelling 16886332 on provider astro proc_lsst.multi INFO: cancelling 16886332 on provider astro 2024-03-07 06:22:34 proc_lsst.multi:146 [INFO] found job 16886337 in provider astro proc_lsst.multi INFO: found job 16886337 in provider astro 2024-03-07 06:22:34 proc_lsst.multi:201 [INFO] cancelling 16886337 on provider astro proc_lsst.multi INFO: cancelling 16886337 on provider astro 2024-03-07 06:22:34 proc_lsst.multi:146 [INFO] found job 16886343 in provider astro proc_lsst.multi INFO: found job 16886343 in provider astro 2024-03-07 06:22:34 proc_lsst.multi:201 [INFO] cancelling 16886343 on provider astro proc_lsst.multi INFO: cancelling 16886343 on provider astro 2024-03-07 06:22:34 proc_lsst.multi:146 [INFO] found job 16886338 in provider astro proc_lsst.multi INFO: found job 16886338 in provider astro 2024-03-07 06:22:34 proc_lsst.multi:201 [INFO] cancelling 16886338 on provider astro proc_lsst.multi INFO: cancelling 16886338 on provider astro 2024-03-07 06:22:34 proc_lsst.multi:146 [INFO] found job 16886352 in provider astro proc_lsst.multi INFO: found job 16886352 in provider astro 2024-03-07 06:22:34 proc_lsst.multi:201 [INFO] cancelling 16886352 on provider astro proc_lsst.multi INFO: cancelling 16886352 on provider astro 2024-03-07 06:22:34 proc_lsst.multi:146 [INFO] found job 16886358 in provider astro proc_lsst.multi INFO: found job 16886358 in provider astro 2024-03-07 06:22:34 proc_lsst.multi:201 [INFO] cancelling 16886358 on provider astro proc_lsst.multi INFO: cancelling 16886358 on provider astro 2024-03-07 06:22:34 proc_lsst.multi:146 [INFO] found job 16886356 in provider astro proc_lsst.multi INFO: found job 16886356 in provider astro 2024-03-07 06:22:34 proc_lsst.multi:201 [INFO] cancelling 16886356 on provider astro proc_lsst.multi INFO: cancelling 16886356 on provider astro 2024-03-07 06:22:34 proc_lsst.multi:146 [INFO] found job 64667 in provider local proc_lsst.multi INFO: found job 64667 in provider local 2024-03-07 06:22:34 proc_lsst.multi:201 [INFO] cancelling 64667 on provider local proc_lsst.multi INFO: cancelling 64667 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 64667 2024-03-07 06:22:34 proc_lsst.multi:146 [INFO] found job 16886342 in provider astro proc_lsst.multi INFO: found job 16886342 in provider astro 2024-03-07 06:22:34 proc_lsst.multi:201 [INFO] cancelling 16886342 on provider astro proc_lsst.multi INFO: cancelling 16886342 on provider astro 2024-03-07 06:22:34 proc_lsst.multi:146 [INFO] found job 16886355 in provider astro proc_lsst.multi INFO: found job 16886355 in provider astro 2024-03-07 06:22:34 proc_lsst.multi:201 [INFO] cancelling 16886355 on provider astro proc_lsst.multi INFO: cancelling 16886355 on provider astro 2024-03-07 06:22:34 proc_lsst.multi:146 [INFO] found job 16886359 in provider astro proc_lsst.multi INFO: found job 16886359 in provider astro 2024-03-07 06:22:34 proc_lsst.multi:201 [INFO] cancelling 16886359 on provider astro proc_lsst.multi INFO: cancelling 16886359 on provider astro 2024-03-07 06:22:34 proc_lsst.multi:146 [INFO] found job 16886353 in provider astro proc_lsst.multi INFO: found job 16886353 in provider astro 2024-03-07 06:22:34 proc_lsst.multi:201 [INFO] cancelling 16886353 on provider astro proc_lsst.multi INFO: cancelling 16886353 on provider astro 2024-03-07 06:22:35 proc_lsst.multi:146 [INFO] found job 16886351 in provider astro proc_lsst.multi INFO: found job 16886351 in provider astro 2024-03-07 06:22:35 proc_lsst.multi:201 [INFO] cancelling 16886351 on provider astro proc_lsst.multi INFO: cancelling 16886351 on provider astro 2024-03-07 06:22:35 proc_lsst.multi:146 [INFO] found job 16886354 in provider astro proc_lsst.multi INFO: found job 16886354 in provider astro 2024-03-07 06:22:35 proc_lsst.multi:201 [INFO] cancelling 16886354 on provider astro proc_lsst.multi INFO: cancelling 16886354 on provider astro 2024-03-07 06:22:35 proc_lsst.multi:146 [INFO] found job 16886344 in provider astro proc_lsst.multi INFO: found job 16886344 in provider astro 2024-03-07 06:22:35 proc_lsst.multi:201 [INFO] cancelling 16886344 on provider astro proc_lsst.multi INFO: cancelling 16886344 on provider astro 2024-03-07 06:22:35 proc_lsst.multi:146 [INFO] found job 16886349 in provider astro proc_lsst.multi INFO: found job 16886349 in provider astro 2024-03-07 06:22:35 proc_lsst.multi:201 [INFO] cancelling 16886349 on provider astro proc_lsst.multi INFO: cancelling 16886349 on provider astro 2024-03-07 06:22:35 proc_lsst.multi:146 [INFO] found job 16886340 in provider astro proc_lsst.multi INFO: found job 16886340 in provider astro 2024-03-07 06:22:35 proc_lsst.multi:201 [INFO] cancelling 16886340 on provider astro proc_lsst.multi INFO: cancelling 16886340 on provider astro 2024-03-07 06:22:35 proc_lsst.multi:146 [INFO] found job 16886350 in provider astro proc_lsst.multi INFO: found job 16886350 in provider astro 2024-03-07 06:22:35 proc_lsst.multi:201 [INFO] cancelling 16886350 on provider astro proc_lsst.multi INFO: cancelling 16886350 on provider astro 2024-03-07 06:22:35 proc_lsst.multi:146 [INFO] found job 16886346 in provider astro proc_lsst.multi INFO: found job 16886346 in provider astro 2024-03-07 06:22:35 proc_lsst.multi:201 [INFO] cancelling 16886346 on provider astro proc_lsst.multi INFO: cancelling 16886346 on provider astro 2024-03-07 06:22:35 proc_lsst.multi:146 [INFO] found job 16886345 in provider astro proc_lsst.multi INFO: found job 16886345 in provider astro 2024-03-07 06:22:35 proc_lsst.multi:201 [INFO] cancelling 16886345 on provider astro proc_lsst.multi INFO: cancelling 16886345 on provider astro 2024-03-07 06:22:35 proc_lsst.multi:146 [INFO] found job 16886341 in provider astro proc_lsst.multi INFO: found job 16886341 in provider astro 2024-03-07 06:22:35 proc_lsst.multi:201 [INFO] cancelling 16886341 on provider astro proc_lsst.multi INFO: cancelling 16886341 on provider astro 2024-03-07 06:22:35 proc_lsst.multi:146 [INFO] found job 16886347 in provider astro proc_lsst.multi INFO: found job 16886347 in provider astro 2024-03-07 06:22:35 proc_lsst.multi:201 [INFO] cancelling 16886347 on provider astro proc_lsst.multi INFO: cancelling 16886347 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-07 06:22:35 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-07 06:22:35 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16886330', '16886331', '16886332', '16886333', '16886334', '16886335', '16886336', '16886337', '16886338', '16886339', '16886340', '16886341', '16886342', '16886343', '16886344', '16886345', '16886346', '16886347', '16886348', '16886349', '16886350', '16886351', '16886352', '16886353', '16886354', '16886355', '16886356', '16886357', '16886358', '16886359', '64667'] proc_lsst.multi INFO: new jobs since last cancel ['16886330', '16886331', '16886332', '16886333', '16886334', '16886335', '16886336', '16886337', '16886338', '16886339', '16886340', '16886341', '16886342', '16886343', '16886344', '16886345', '16886346', '16886347', '16886348', '16886349', '16886350', '16886351', '16886352', '16886353', '16886354', '16886355', '16886356', '16886357', '16886358', '16886359', '64667'] 2024-03-07 06:22:35 proc_lsst.multi:146 [INFO] found job 16886330 in provider astro proc_lsst.multi INFO: found job 16886330 in provider astro 2024-03-07 06:22:35 proc_lsst.multi:201 [INFO] cancelling 16886330 on provider astro proc_lsst.multi INFO: cancelling 16886330 on provider astro 2024-03-07 06:22:35 proc_lsst.multi:146 [INFO] found job 16886331 in provider astro proc_lsst.multi INFO: found job 16886331 in provider astro 2024-03-07 06:22:35 proc_lsst.multi:201 [INFO] cancelling 16886331 on provider astro proc_lsst.multi INFO: cancelling 16886331 on provider astro 2024-03-07 06:22:36 proc_lsst.multi:146 [INFO] found job 16886332 in provider astro proc_lsst.multi INFO: found job 16886332 in provider astro 2024-03-07 06:22:36 proc_lsst.multi:201 [INFO] cancelling 16886332 on provider astro proc_lsst.multi INFO: cancelling 16886332 on provider astro 2024-03-07 06:22:36 proc_lsst.multi:146 [INFO] found job 16886333 in provider astro proc_lsst.multi INFO: found job 16886333 in provider astro 2024-03-07 06:22:36 proc_lsst.multi:201 [INFO] cancelling 16886333 on provider astro proc_lsst.multi INFO: cancelling 16886333 on provider astro 2024-03-07 06:22:36 proc_lsst.multi:146 [INFO] found job 16886334 in provider astro proc_lsst.multi INFO: found job 16886334 in provider astro 2024-03-07 06:22:36 proc_lsst.multi:201 [INFO] cancelling 16886334 on provider astro proc_lsst.multi INFO: cancelling 16886334 on provider astro 2024-03-07 06:22:36 proc_lsst.multi:146 [INFO] found job 16886335 in provider astro proc_lsst.multi INFO: found job 16886335 in provider astro 2024-03-07 06:22:36 proc_lsst.multi:201 [INFO] cancelling 16886335 on provider astro proc_lsst.multi INFO: cancelling 16886335 on provider astro 2024-03-07 06:22:36 proc_lsst.multi:146 [INFO] found job 16886336 in provider astro proc_lsst.multi INFO: found job 16886336 in provider astro 2024-03-07 06:22:36 proc_lsst.multi:201 [INFO] cancelling 16886336 on provider astro proc_lsst.multi INFO: cancelling 16886336 on provider astro 2024-03-07 06:22:36 proc_lsst.multi:146 [INFO] found job 16886337 in provider astro proc_lsst.multi INFO: found job 16886337 in provider astro 2024-03-07 06:22:36 proc_lsst.multi:201 [INFO] cancelling 16886337 on provider astro proc_lsst.multi INFO: cancelling 16886337 on provider astro 2024-03-07 06:22:36 proc_lsst.multi:146 [INFO] found job 16886338 in provider astro proc_lsst.multi INFO: found job 16886338 in provider astro 2024-03-07 06:22:36 proc_lsst.multi:201 [INFO] cancelling 16886338 on provider astro proc_lsst.multi INFO: cancelling 16886338 on provider astro 2024-03-07 06:22:36 proc_lsst.multi:146 [INFO] found job 16886339 in provider astro proc_lsst.multi INFO: found job 16886339 in provider astro 2024-03-07 06:22:36 proc_lsst.multi:201 [INFO] cancelling 16886339 on provider astro proc_lsst.multi INFO: cancelling 16886339 on provider astro 2024-03-07 06:22:36 proc_lsst.multi:146 [INFO] found job 16886340 in provider astro proc_lsst.multi INFO: found job 16886340 in provider astro 2024-03-07 06:22:36 proc_lsst.multi:201 [INFO] cancelling 16886340 on provider astro proc_lsst.multi INFO: cancelling 16886340 on provider astro 2024-03-07 06:22:36 proc_lsst.multi:146 [INFO] found job 16886341 in provider astro proc_lsst.multi INFO: found job 16886341 in provider astro 2024-03-07 06:22:36 proc_lsst.multi:201 [INFO] cancelling 16886341 on provider astro proc_lsst.multi INFO: cancelling 16886341 on provider astro 2024-03-07 06:22:36 proc_lsst.multi:146 [INFO] found job 16886342 in provider astro proc_lsst.multi INFO: found job 16886342 in provider astro 2024-03-07 06:22:36 proc_lsst.multi:201 [INFO] cancelling 16886342 on provider astro proc_lsst.multi INFO: cancelling 16886342 on provider astro 2024-03-07 06:22:37 proc_lsst.multi:146 [INFO] found job 16886343 in provider astro proc_lsst.multi INFO: found job 16886343 in provider astro 2024-03-07 06:22:37 proc_lsst.multi:201 [INFO] cancelling 16886343 on provider astro proc_lsst.multi INFO: cancelling 16886343 on provider astro 2024-03-07 06:22:37 proc_lsst.multi:146 [INFO] found job 16886344 in provider astro proc_lsst.multi INFO: found job 16886344 in provider astro 2024-03-07 06:22:37 proc_lsst.multi:201 [INFO] cancelling 16886344 on provider astro proc_lsst.multi INFO: cancelling 16886344 on provider astro 2024-03-07 06:22:37 proc_lsst.multi:146 [INFO] found job 16886345 in provider astro proc_lsst.multi INFO: found job 16886345 in provider astro 2024-03-07 06:22:37 proc_lsst.multi:201 [INFO] cancelling 16886345 on provider astro proc_lsst.multi INFO: cancelling 16886345 on provider astro 2024-03-07 06:22:37 proc_lsst.multi:146 [INFO] found job 16886346 in provider astro proc_lsst.multi INFO: found job 16886346 in provider astro 2024-03-07 06:22:37 proc_lsst.multi:201 [INFO] cancelling 16886346 on provider astro proc_lsst.multi INFO: cancelling 16886346 on provider astro 2024-03-07 06:22:37 proc_lsst.multi:146 [INFO] found job 16886347 in provider astro proc_lsst.multi INFO: found job 16886347 in provider astro 2024-03-07 06:22:37 proc_lsst.multi:201 [INFO] cancelling 16886347 on provider astro proc_lsst.multi INFO: cancelling 16886347 on provider astro 2024-03-07 06:22:37 proc_lsst.multi:146 [INFO] found job 16886348 in provider astro proc_lsst.multi INFO: found job 16886348 in provider astro 2024-03-07 06:22:37 proc_lsst.multi:201 [INFO] cancelling 16886348 on provider astro proc_lsst.multi INFO: cancelling 16886348 on provider astro 2024-03-07 06:22:37 proc_lsst.multi:146 [INFO] found job 16886349 in provider astro proc_lsst.multi INFO: found job 16886349 in provider astro 2024-03-07 06:22:37 proc_lsst.multi:201 [INFO] cancelling 16886349 on provider astro proc_lsst.multi INFO: cancelling 16886349 on provider astro 2024-03-07 06:22:37 proc_lsst.multi:146 [INFO] found job 16886350 in provider astro proc_lsst.multi INFO: found job 16886350 in provider astro 2024-03-07 06:22:37 proc_lsst.multi:201 [INFO] cancelling 16886350 on provider astro proc_lsst.multi INFO: cancelling 16886350 on provider astro 2024-03-07 06:22:37 proc_lsst.multi:146 [INFO] found job 16886351 in provider astro proc_lsst.multi INFO: found job 16886351 in provider astro 2024-03-07 06:22:37 proc_lsst.multi:201 [INFO] cancelling 16886351 on provider astro proc_lsst.multi INFO: cancelling 16886351 on provider astro 2024-03-07 06:22:39 proc_lsst.multi:146 [INFO] found job 16886352 in provider astro proc_lsst.multi INFO: found job 16886352 in provider astro 2024-03-07 06:22:39 proc_lsst.multi:201 [INFO] cancelling 16886352 on provider astro proc_lsst.multi INFO: cancelling 16886352 on provider astro 2024-03-07 06:22:39 proc_lsst.multi:146 [INFO] found job 16886353 in provider astro proc_lsst.multi INFO: found job 16886353 in provider astro 2024-03-07 06:22:39 proc_lsst.multi:201 [INFO] cancelling 16886353 on provider astro proc_lsst.multi INFO: cancelling 16886353 on provider astro 2024-03-07 06:22:40 proc_lsst.multi:146 [INFO] found job 16886354 in provider astro proc_lsst.multi INFO: found job 16886354 in provider astro 2024-03-07 06:22:40 proc_lsst.multi:201 [INFO] cancelling 16886354 on provider astro proc_lsst.multi INFO: cancelling 16886354 on provider astro 2024-03-07 06:22:40 proc_lsst.multi:146 [INFO] found job 16886355 in provider astro proc_lsst.multi INFO: found job 16886355 in provider astro 2024-03-07 06:22:40 proc_lsst.multi:201 [INFO] cancelling 16886355 on provider astro proc_lsst.multi INFO: cancelling 16886355 on provider astro 2024-03-07 06:22:40 proc_lsst.multi:146 [INFO] found job 16886356 in provider astro proc_lsst.multi INFO: found job 16886356 in provider astro 2024-03-07 06:22:40 proc_lsst.multi:201 [INFO] cancelling 16886356 on provider astro proc_lsst.multi INFO: cancelling 16886356 on provider astro 2024-03-07 06:22:40 proc_lsst.multi:146 [INFO] found job 16886357 in provider astro proc_lsst.multi INFO: found job 16886357 in provider astro 2024-03-07 06:22:40 proc_lsst.multi:201 [INFO] cancelling 16886357 on provider astro proc_lsst.multi INFO: cancelling 16886357 on provider astro 2024-03-07 06:22:40 proc_lsst.multi:146 [INFO] found job 16886358 in provider astro proc_lsst.multi INFO: found job 16886358 in provider astro 2024-03-07 06:22:40 proc_lsst.multi:201 [INFO] cancelling 16886358 on provider astro proc_lsst.multi INFO: cancelling 16886358 on provider astro 2024-03-07 06:22:41 proc_lsst.multi:146 [INFO] found job 16886359 in provider astro proc_lsst.multi INFO: found job 16886359 in provider astro 2024-03-07 06:22:41 proc_lsst.multi:201 [INFO] cancelling 16886359 on provider astro proc_lsst.multi INFO: cancelling 16886359 on provider astro 2024-03-07 06:22:41 proc_lsst.multi:146 [INFO] found job 64667 in provider local proc_lsst.multi INFO: found job 64667 in provider local 2024-03-07 06:22:41 proc_lsst.multi:201 [INFO] cancelling 64667 on provider local proc_lsst.multi INFO: cancelling 64667 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 64667 parsl.providers.local.local WARNING: Failed to kill PID: 64667 and child processes on local 2024-03-07 06:22:42 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 109.1089 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20220822_B1m_science#step0_20240307T062031Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 109.1153 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 128.9385 seconds; current memory usage: 0.270 Gibyte, delta: 0.134 Gibyte, peak delta: 0.136 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.272 Gibyte (largest child process) Run Id: None Run Name: DEEP_20220822_B1m_science#step0_20240307T062031Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up