lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.3296 seconds; current memory usage: 0.178 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0025 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0060 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/pipeline_orig.qgraph DEEP/20220827/B1l/science#step0/20240307T191019Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.5736 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 6.6774 seconds; current memory usage: 0.270 Gibyte, delta: 0.092 Gibyte, peak delta: 0.079 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.259 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0100 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.259 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 186 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20220827_B1l_science#step0_20240307T191019Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0496 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.259 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 188 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.2310 seconds; current memory usage: 0.271 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.271 Gibyte (main), 0.259 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 0a757521-cba3-4fb4-8821-30185c48597a parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-07 19:10:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-07 19:10:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 19:10:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-07 19:10:42 proc_lsst.multi:170 [INFO] [multi] job_id 42497 proc_lsst.multi INFO: [multi] job_id 42497 2024-03-07 19:10:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:42 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 42497 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 42497 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 42497 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 7f493a23e49e8b037795659f7beca8e0 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943034c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/08d495c4-e1b3-40f8-9a3a-211bdf662c3b_isrForCrosstalkSources_1126400_47.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/08d495c4-e1b3-40f8-9a3a-211bdf662c3b_isrForCrosstalkSources_1126400_47.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 33f9c18ca9670accd67c3871556cfcde parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094303880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/cadb8287-ecfc-486f-80c8-d7c4a3a54657_isrForCrosstalkSources_1126400_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/cadb8287-ecfc-486f-80c8-d7c4a3a54657_isrForCrosstalkSources_1126400_4.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash e39b8f0ab777f2cb106b929a304a4d24 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094303b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/44bd78df-6e74-42d3-b57e-f86ffe8be357_isrForCrosstalkSources_1126402_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/44bd78df-6e74-42d3-b57e-f86ffe8be357_isrForCrosstalkSources_1126402_11.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 405f3388ad753747ab760c3ed04da969 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943039c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/09c9a1be-af8d-4adc-bd6a-1ead23050f1e_isrForCrosstalkSources_1126401_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/09c9a1be-af8d-4adc-bd6a-1ead23050f1e_isrForCrosstalkSources_1126401_25.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 4e3fa13735476dfbd801ef5a4c024f33 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094303a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/090ae0d2-9469-4976-a6a6-eb9307b3c459_isrForCrosstalkSources_1126401_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/090ae0d2-9469-4976-a6a6-eb9307b3c459_isrForCrosstalkSources_1126401_48.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 867d19ca01b576f09c65ebe7aa6ee345 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094303c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/acfbde09-ba8e-46af-956d-00bebf329e6b_isrForCrosstalkSources_1126401_27.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/acfbde09-ba8e-46af-956d-00bebf329e6b_isrForCrosstalkSources_1126401_27.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash fd2a2f8cc80be75b8731b2f8d0f930af parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943037e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/cc06a8de-8202-4229-bdc0-34ff2f78d894_isrForCrosstalkSources_1126402_20.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/cc06a8de-8202-4229-bdc0-34ff2f78d894_isrForCrosstalkSources_1126402_20.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 1bae5548b30035a365b371d847f065cc parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094303e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/6348e86a-37b2-4bb1-b06f-e7858f7e1c71_isrForCrosstalkSources_1126401_9.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/6348e86a-37b2-4bb1-b06f-e7858f7e1c71_isrForCrosstalkSources_1126401_9.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash e34425344caedbe4f2e9be7b6860d87f parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094302de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d30b5e8c-4d0b-4eb8-a9b5-c959733ce81e_isrForCrosstalkSources_1126400_21.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d30b5e8c-4d0b-4eb8-a9b5-c959733ce81e_isrForCrosstalkSources_1126400_21.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash beed196ce6b81d4896db0fa3bab99502 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943036a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/09e88ef5-1859-4991-b073-0938209d738c_isrForCrosstalkSources_1126401_40.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/09e88ef5-1859-4991-b073-0938209d738c_isrForCrosstalkSources_1126401_40.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 5f152782a4cb9e3c6f1ed82ef230bcfb parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094303ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/950b505d-1fd1-409b-aa8c-98632c9d591f_isrForCrosstalkSources_1126400_54.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/950b505d-1fd1-409b-aa8c-98632c9d591f_isrForCrosstalkSources_1126400_54.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash b9d97a2e7dd9e28cc0f0655876f6545d parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094303ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/f7b883d5-8741-447e-b13a-d5512dafabef_isrForCrosstalkSources_1126402_43.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/f7b883d5-8741-447e-b13a-d5512dafabef_isrForCrosstalkSources_1126402_43.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 4eb3505095230a835bdd15861c7915e0 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943440e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/a88865d8-dfca-4eb7-b899-3a7da5fe443d_isrForCrosstalkSources_1126401_45.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/a88865d8-dfca-4eb7-b899-3a7da5fe443d_isrForCrosstalkSources_1126401_45.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 2f596777bd971dec1697c38f49aafcb8 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/6eb989e1-afe0-4019-b123-88b6975b6340_isrForCrosstalkSources_1126400_17.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/6eb989e1-afe0-4019-b123-88b6975b6340_isrForCrosstalkSources_1126400_17.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 78cd12fea61e16f2440aeae11c408377 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943445e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/0ff7ad2a-0d19-4e8c-9c06-8d77556f79c4_isrForCrosstalkSources_1126402_33.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/0ff7ad2a-0d19-4e8c-9c06-8d77556f79c4_isrForCrosstalkSources_1126402_33.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 0841d32994dafe263c68894a6d7b7d1c parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943442c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/8744cf61-89ef-4362-ae82-f5453c5716f7_isrForCrosstalkSources_1126400_40.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/8744cf61-89ef-4362-ae82-f5453c5716f7_isrForCrosstalkSources_1126400_40.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 81fe8caf5a82b1c6eb063788ecd7a053 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/384226e6-f30f-436f-b623-4fc900aa84a5_isrForCrosstalkSources_1126401_39.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/384226e6-f30f-436f-b623-4fc900aa84a5_isrForCrosstalkSources_1126401_39.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 26ea9e1089af403877836876d9384497 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943444a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/27717ee2-1dff-4ff6-bde9-bdd6f33fdc78_isrForCrosstalkSources_1126400_61.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/27717ee2-1dff-4ff6-bde9-bdd6f33fdc78_isrForCrosstalkSources_1126400_61.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 8b355a8d8ce938af4aab0f7bba0ca6b6 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/79c1dc82-6267-4316-912c-8bd4897c8169_isrForCrosstalkSources_1126401_28.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/79c1dc82-6267-4316-912c-8bd4897c8169_isrForCrosstalkSources_1126401_28.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 34b47d43dff31fb85ad58b9170d8781f parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943447c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/a69fe013-b834-491f-97f9-b93c0e9f1d19_isrForCrosstalkSources_1126402_62.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/a69fe013-b834-491f-97f9-b93c0e9f1d19_isrForCrosstalkSources_1126402_62.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 60286cc8b02241cb0f7c3e54ee327364 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/af60ada3-fa34-4744-8f10-b2f0936b45f1_isrForCrosstalkSources_1126402_24.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/af60ada3-fa34-4744-8f10-b2f0936b45f1_isrForCrosstalkSources_1126402_24.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 0109625180f9332174a05cc3a3018083 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943449a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d3888e6f-28e3-4360-b224-d128efd8030a_isrForCrosstalkSources_1126400_23.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d3888e6f-28e3-4360-b224-d128efd8030a_isrForCrosstalkSources_1126400_23.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 0f07466577330841093198cbcd56670b parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/2ac9160e-cda0-4afc-a590-1ac0f739a258_isrForCrosstalkSources_1126400_41.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/2ac9160e-cda0-4afc-a590-1ac0f739a258_isrForCrosstalkSources_1126400_41.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash dc34405ee8caa4d385a5d56d1aedfa7f parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1d024acc-5a13-434c-8e93-1613be78f5dd_isrForCrosstalkSources_1126402_51.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1d024acc-5a13-434c-8e93-1613be78f5dd_isrForCrosstalkSources_1126402_51.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 3d7638c5e5f36523c1f060f1cbb1eeab parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/528f2f35-d740-4be0-875a-a205ef019930_isrForCrosstalkSources_1126402_52.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/528f2f35-d740-4be0-875a-a205ef019930_isrForCrosstalkSources_1126402_52.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash d77058671d01c9674ef5c2ab726139f8 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/5fbc53da-00dd-4ef4-8271-ea147c7d0692_isrForCrosstalkSources_1126400_37.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/5fbc53da-00dd-4ef4-8271-ea147c7d0692_isrForCrosstalkSources_1126400_37.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 9ea5b06c1a086927228bff5d64570efb parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/099868cc-903d-4b38-b52d-4d1a209cee86_isrForCrosstalkSources_1126400_28.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/099868cc-903d-4b38-b52d-4d1a209cee86_isrForCrosstalkSources_1126400_28.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 202e05bbe8d03846cfaadd7333f9a2a2 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/48e58e4f-32b1-4ccb-8225-81587f9c8de4_isrForCrosstalkSources_1126401_60.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/48e58e4f-32b1-4ccb-8225-81587f9c8de4_isrForCrosstalkSources_1126401_60.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash cc867c18245ec0b33bc3cdc0a3215f3c parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/4256510f-e43d-4421-9257-230f8cd64f36_isrForCrosstalkSources_1126402_53.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/4256510f-e43d-4421-9257-230f8cd64f36_isrForCrosstalkSources_1126402_53.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 0c63c8ea0ac42539045ba2664e5e31d6 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cdae455e-d342-4022-959a-e8b03d06230c_isrForCrosstalkSources_1126401_3.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cdae455e-d342-4022-959a-e8b03d06230c_isrForCrosstalkSources_1126401_3.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 400e4d7fdff95c7859cc450db9cbfc10 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/5aa4efca-7897-44fc-b9d5-b28c6235fc5a_isrForCrosstalkSources_1126401_18.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/5aa4efca-7897-44fc-b9d5-b28c6235fc5a_isrForCrosstalkSources_1126401_18.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash ab162b52f552fad6a1394e708df4913b parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/cf1d847e-96f0-44c3-9593-6b5c9e25960a_isrForCrosstalkSources_1126402_45.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/cf1d847e-96f0-44c3-9593-6b5c9e25960a_isrForCrosstalkSources_1126402_45.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash c6a03521238276269936066d760097da parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/6ab5d426-bcac-4d5a-acb8-b11d21c16b6b_isrForCrosstalkSources_1126401_41.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/6ab5d426-bcac-4d5a-acb8-b11d21c16b6b_isrForCrosstalkSources_1126401_41.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 635ae86deeff6e9952b8844716c324a4 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/31845923-9ebe-41c9-b0ad-f42fd26dd003_isrForCrosstalkSources_1126402_36.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/31845923-9ebe-41c9-b0ad-f42fd26dd003_isrForCrosstalkSources_1126402_36.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 6f9fdf4fe12c318a066e0addb38f43c0 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/68b91aca-64bb-40bb-8269-28cfd1ead066_isrForCrosstalkSources_1126400_35.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/68b91aca-64bb-40bb-8269-28cfd1ead066_isrForCrosstalkSources_1126400_35.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash b98df085da5226c18d847f2f6d0916a4 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943451c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ac5e93d0-a697-4af4-95e9-5cc6f3952891_isrForCrosstalkSources_1126400_1.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ac5e93d0-a697-4af4-95e9-5cc6f3952891_isrForCrosstalkSources_1126400_1.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 8fe8998b577f534416b939d72b54ccec parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/006d64bf-4dc0-4a2d-8e36-0fbc6f7fe4fa_isrForCrosstalkSources_1126402_54.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/006d64bf-4dc0-4a2d-8e36-0fbc6f7fe4fa_isrForCrosstalkSources_1126402_54.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 99b261e1d18b9ea445687344e7ccd32c parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/e607ac5c-b8a0-42f4-8ca4-2c2d7b8d0489_isrForCrosstalkSources_1126400_12.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/e607ac5c-b8a0-42f4-8ca4-2c2d7b8d0489_isrForCrosstalkSources_1126400_12.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash ac659dd2ea07afa8845feb9feb798871 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943453a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/a653ab59-12cb-4167-b8f4-f20474ab7b01_isrForCrosstalkSources_1126401_30.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/a653ab59-12cb-4167-b8f4-f20474ab7b01_isrForCrosstalkSources_1126401_30.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash f8ec515438886f8753ba6eff575a93da parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/5a5a2f1f-6cc8-4276-b208-9c7f24f402a1_isrForCrosstalkSources_1126402_42.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/5a5a2f1f-6cc8-4276-b208-9c7f24f402a1_isrForCrosstalkSources_1126402_42.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 42e6d2303e1d11fd90e9cd930244ea18 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943454e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ed6e664a-79e4-4292-98af-449194a2719f_isrForCrosstalkSources_1126400_51.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ed6e664a-79e4-4292-98af-449194a2719f_isrForCrosstalkSources_1126400_51.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 431daee726fe304f8d31e4d04399449d parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/22d7be72-01d4-4276-a9b8-c1a7a954aeb2_isrForCrosstalkSources_1126402_48.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/22d7be72-01d4-4276-a9b8-c1a7a954aeb2_isrForCrosstalkSources_1126402_48.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash f88cc46f29e9037ba9bcc92dc9c379db parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1f797ac3-94e1-42de-8952-1266e4aa7776_isrForCrosstalkSources_1126402_2.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1f797ac3-94e1-42de-8952-1266e4aa7776_isrForCrosstalkSources_1126402_2.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 263ac898bf28a2a8884d2eb350c6c5d7 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943456c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/3923fb1b-5b19-483e-a3d8-c2d90bf3b071_isrForCrosstalkSources_1126401_44.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/3923fb1b-5b19-483e-a3d8-c2d90bf3b071_isrForCrosstalkSources_1126401_44.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash a687db940405a7216d7f2fca39d18400 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cbf23c54-1eea-4fc0-9677-56f07ae1563e_isrForCrosstalkSources_1126401_57.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cbf23c54-1eea-4fc0-9677-56f07ae1563e_isrForCrosstalkSources_1126401_57.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 1709d152ad0fddacd0f3db1bf319adfe parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/eacadb65-b819-4e4b-b57b-328d50aaf7dc_isrForCrosstalkSources_1126400_42.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/eacadb65-b819-4e4b-b57b-328d50aaf7dc_isrForCrosstalkSources_1126400_42.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash a7fa8aa6c8496036ecc7e128adf33a0b parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/dfdbbcbd-f674-44f6-973d-aca2f9b5ec8c_isrForCrosstalkSources_1126401_54.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/dfdbbcbd-f674-44f6-973d-aca2f9b5ec8c_isrForCrosstalkSources_1126401_54.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash a8c6c7ef9931448cabde4356d68e679e parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943458a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/447b5c4f-8131-4ff9-8714-83f55b89a6a2_isrForCrosstalkSources_1126401_46.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/447b5c4f-8131-4ff9-8714-83f55b89a6a2_isrForCrosstalkSources_1126401_46.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 06bbbd186046b6e535d8ea074d0b00ff parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943459e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/05ee01ea-45ff-4c8a-b2fd-81bb0f419a30_isrForCrosstalkSources_1126402_9.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/05ee01ea-45ff-4c8a-b2fd-81bb0f419a30_isrForCrosstalkSources_1126402_9.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash d934f75066a1b895de36e049e56d35cb parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/93ea9278-80ab-4e23-a8f1-897ff0716b47_isrForCrosstalkSources_1126400_9.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/93ea9278-80ab-4e23-a8f1-897ff0716b47_isrForCrosstalkSources_1126400_9.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 1121a35e14bd2fc7df8954a93be4b785 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1c092bcf-031b-49b4-869e-34ec3536d0de_isrForCrosstalkSources_1126402_27.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1c092bcf-031b-49b4-869e-34ec3536d0de_isrForCrosstalkSources_1126402_27.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 03176aad0238c064332d7f046f5a9172 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fb727ed7-ca16-4831-b859-8397adc4c2c2_isrForCrosstalkSources_1126401_21.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fb727ed7-ca16-4831-b859-8397adc4c2c2_isrForCrosstalkSources_1126401_21.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash efdd0ac20d6a98a82dd3c17b5f2ee7d6 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/5e4f06f5-a364-4d78-b125-9188d4d644b3_isrForCrosstalkSources_1126401_7.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/5e4f06f5-a364-4d78-b125-9188d4d644b3_isrForCrosstalkSources_1126401_7.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 4a1458fd40b6c1c0329bbcf57bd736e4 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/3d916b70-2b81-4813-9fa8-0f950b25c3e8_isrForCrosstalkSources_1126402_19.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/3d916b70-2b81-4813-9fa8-0f950b25c3e8_isrForCrosstalkSources_1126402_19.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash fc7eba193d5f42085f5763b6ca8cb019 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/f423c78f-5423-46de-a377-9df0e8b47d65_isrForCrosstalkSources_1126401_12.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/f423c78f-5423-46de-a377-9df0e8b47d65_isrForCrosstalkSources_1126401_12.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 9f1565d40eb43d452ee16946b68d8784 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/c01bc846-60ef-4a47-8e04-b08902c4da2c_isrForCrosstalkSources_1126400_48.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/c01bc846-60ef-4a47-8e04-b08902c4da2c_isrForCrosstalkSources_1126400_48.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 8fd17f6bb6c011fa7f1a1dc31bc9c95d parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/796a034b-2b2f-49f3-900f-c1899c74a0e1_isrForCrosstalkSources_1126401_49.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/796a034b-2b2f-49f3-900f-c1899c74a0e1_isrForCrosstalkSources_1126401_49.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 6acc1e216abfac6d040b1494e9a1eec3 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094345ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/dd011805-93ea-440e-849e-7fbb2b7c11f5_isrForCrosstalkSources_1126400_31.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/dd011805-93ea-440e-849e-7fbb2b7c11f5_isrForCrosstalkSources_1126400_31.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 4fe0a50225ce8a1d13d32365bc8bdb09 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/3b3c3dc3-ed50-4a07-bae6-c90e5463a659_isrForCrosstalkSources_1126402_8.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/3b3c3dc3-ed50-4a07-bae6-c90e5463a659_isrForCrosstalkSources_1126402_8.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 0ac1c5311ca7ee5a050b677d1f21f56d parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943460c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/824cbe35-8c21-4f40-a881-7adbbd087e82_isrForCrosstalkSources_1126400_36.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/824cbe35-8c21-4f40-a881-7adbbd087e82_isrForCrosstalkSources_1126400_36.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash c20407558b21c5c2ea48915c4c0cf15a parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/621b695d-2f29-48c3-bef5-9726653b4073_isrForCrosstalkSources_1126401_35.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/621b695d-2f29-48c3-bef5-9726653b4073_isrForCrosstalkSources_1126401_35.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 4c2471da8558d39a8a54573c737ef45a parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943462a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/034c916f-5818-4b0c-ac78-a4a0d9cf8445_isrForCrosstalkSources_1126401_31.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/034c916f-5818-4b0c-ac78-a4a0d9cf8445_isrForCrosstalkSources_1126401_31.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 96392a46d7deba90bef26922686fbd4e parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/3a9b3b4a-f9ae-4c94-949b-e3a97d45fffc_isrForCrosstalkSources_1126401_52.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/3a9b3b4a-f9ae-4c94-949b-e3a97d45fffc_isrForCrosstalkSources_1126401_52.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 5744b12f5658d2485e10ab4fa241b2f3 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b64f9fdc-c36d-41c0-a2a4-3ae0fcd3b67f_isrForCrosstalkSources_1126402_35.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b64f9fdc-c36d-41c0-a2a4-3ae0fcd3b67f_isrForCrosstalkSources_1126402_35.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash a0afcaccf46054405244e50f48592e57 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943463e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/61f20329-fdb8-4620-9163-696e12430594_isrForCrosstalkSources_1126401_5.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/61f20329-fdb8-4620-9163-696e12430594_isrForCrosstalkSources_1126401_5.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash e31407fdf9f8eaccdce38abb4eeba08e parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/8eae20ba-7256-4748-94df-cc1184025650_isrForCrosstalkSources_1126402_50.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/8eae20ba-7256-4748-94df-cc1184025650_isrForCrosstalkSources_1126402_50.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 9a5bb3788537a4c70e673a92d49adcd4 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943465c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/81c6e095-c13f-42fe-b86d-77e0ed1e88a3_isrForCrosstalkSources_1126401_1.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/81c6e095-c13f-42fe-b86d-77e0ed1e88a3_isrForCrosstalkSources_1126401_1.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 59501d9047be181e9dafe52bb7b74727 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/e826bc89-0019-44db-b0a7-f21b12c06ba1_isrForCrosstalkSources_1126401_2.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/e826bc89-0019-44db-b0a7-f21b12c06ba1_isrForCrosstalkSources_1126401_2.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash e6e89493b8421763b0f27cab0cc45087 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cd1bbf6d-7c7c-48ef-abf7-936bd44a6d0e_isrForCrosstalkSources_1126401_47.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cd1bbf6d-7c7c-48ef-abf7-936bd44a6d0e_isrForCrosstalkSources_1126401_47.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 232b419f3f3fc0cecd50559d606b2538 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ccb883c1-356a-4815-a3df-4a4884e029e0_isrForCrosstalkSources_1126402_22.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ccb883c1-356a-4815-a3df-4a4884e029e0_isrForCrosstalkSources_1126402_22.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash b35d458bba01c58b4077f6fd0d5b2018 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943467a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/16116073-2467-49d4-9f35-33c0dae62c19_isrForCrosstalkSources_1126402_31.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/16116073-2467-49d4-9f35-33c0dae62c19_isrForCrosstalkSources_1126402_31.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 9a24857a4e5bead499a1345a977ed569 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943468e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/f3008da6-1da6-44ce-9c8e-befe01bcca12_isrForCrosstalkSources_1126400_29.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/f3008da6-1da6-44ce-9c8e-befe01bcca12_isrForCrosstalkSources_1126400_29.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash ae09f11a068a884b7a76f85b95f55e9e parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/698362b7-9080-4138-8555-62f1f98236eb_isrForCrosstalkSources_1126400_2.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/698362b7-9080-4138-8555-62f1f98236eb_isrForCrosstalkSources_1126400_2.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 424c3ceaad67bc444aea935d95c70da7 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/7cc72bf7-9cdc-4574-802c-3018c179dd09_isrForCrosstalkSources_1126400_30.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/7cc72bf7-9cdc-4574-802c-3018c179dd09_isrForCrosstalkSources_1126400_30.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash a0f7795858ef697ad33c5c60496c9853 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/38a4f24d-813d-4d85-8292-d51d0daf6e93_isrForCrosstalkSources_1126402_47.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/38a4f24d-813d-4d85-8292-d51d0daf6e93_isrForCrosstalkSources_1126402_47.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 0d1e18833af064b67c6e9a62a1dfe4e0 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/caac1eee-a6b6-48b4-b276-392ce6969afa_isrForCrosstalkSources_1126400_59.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/caac1eee-a6b6-48b4-b276-392ce6969afa_isrForCrosstalkSources_1126400_59.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 9ac6ff2911cad48812c6ee2568628b4a parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/754bdbb2-7ed0-4e2f-9496-39e593bcf33c_isrForCrosstalkSources_1126402_34.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/754bdbb2-7ed0-4e2f-9496-39e593bcf33c_isrForCrosstalkSources_1126402_34.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 6cd121f486d689c27abb5a9a7e9e2cee parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/b8d5a844-7dd8-4deb-b548-0f73ac63954b_isrForCrosstalkSources_1126400_18.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/b8d5a844-7dd8-4deb-b548-0f73ac63954b_isrForCrosstalkSources_1126400_18.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash bb988793fa8c05c3be218ff4337b8ea0 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/0d7f0945-f031-4b5e-b3a1-92ffe37973e7_isrForCrosstalkSources_1126400_33.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/0d7f0945-f031-4b5e-b3a1-92ffe37973e7_isrForCrosstalkSources_1126400_33.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 8f9902006a5d8f14d709a1a58041984f parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ae6bb2c5-d585-4a15-81f9-9b34c5273e0a_isrForCrosstalkSources_1126402_49.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ae6bb2c5-d585-4a15-81f9-9b34c5273e0a_isrForCrosstalkSources_1126402_49.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash d238909265c9955da7d7dc645df24848 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/413c8a5c-3732-46d9-bf59-adc79c415004_isrForCrosstalkSources_1126402_38.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/413c8a5c-3732-46d9-bf59-adc79c415004_isrForCrosstalkSources_1126402_38.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 2173de0aef4857bbafb3a59d2427676d parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/42e48eff-19aa-484a-b1e5-0dbebc9aaa83_isrForCrosstalkSources_1126402_39.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/42e48eff-19aa-484a-b1e5-0dbebc9aaa83_isrForCrosstalkSources_1126402_39.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 5b1effa071b3f70127cd4bed6942e4cb parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2f48c7c8-4544-4ec0-9b56-5844e6ab10db_isrForCrosstalkSources_1126402_21.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2f48c7c8-4544-4ec0-9b56-5844e6ab10db_isrForCrosstalkSources_1126402_21.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash e5854a8dd130aa4dc1f92df0e67b355d parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094346fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/c8b123ec-91be-4200-8997-1d0d1cd544b8_isrForCrosstalkSources_1126401_14.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/c8b123ec-91be-4200-8997-1d0d1cd544b8_isrForCrosstalkSources_1126401_14.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash c3af98fc769fab058a95e70738ea85e7 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/7fe262a3-96d9-47ef-bac2-a9c165f12956_isrForCrosstalkSources_1126400_38.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/7fe262a3-96d9-47ef-bac2-a9c165f12956_isrForCrosstalkSources_1126400_38.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash cda25871978fab921d4ad27177a61897 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/eb1fc4cf-bdef-4bdd-b1ad-754904d5f3cd_isrForCrosstalkSources_1126401_24.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/eb1fc4cf-bdef-4bdd-b1ad-754904d5f3cd_isrForCrosstalkSources_1126401_24.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash b6c1273c26eada7179d16bbca2cca814 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/d1a68a75-3908-4b8b-af82-74de954bfe67_isrForCrosstalkSources_1126401_50.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/d1a68a75-3908-4b8b-af82-74de954bfe67_isrForCrosstalkSources_1126401_50.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash fd2bfca7d18acf6b47a8c77be40cb986 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943471a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/aa79c2b0-f917-426d-8125-4e3d393a0162_isrForCrosstalkSources_1126402_26.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/aa79c2b0-f917-426d-8125-4e3d393a0162_isrForCrosstalkSources_1126402_26.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 16310f6b52f62a6154d33d35412066f2 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943472e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/83f5f68d-9fb2-4811-a6df-0c519379a04f_isrForCrosstalkSources_1126402_28.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/83f5f68d-9fb2-4811-a6df-0c519379a04f_isrForCrosstalkSources_1126402_28.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash c8c190034c5fb8e600728d1f51076ec4 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/8b315f22-bc98-4177-a07f-bf4a5de4591f_isrForCrosstalkSources_1126400_24.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/8b315f22-bc98-4177-a07f-bf4a5de4591f_isrForCrosstalkSources_1126400_24.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash fbf3386c10d637c85fdd7fcea15cdd20 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d6971f35-ca0a-4322-a575-d803854781b7_isrForCrosstalkSources_1126400_58.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d6971f35-ca0a-4322-a575-d803854781b7_isrForCrosstalkSources_1126400_58.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 9d788cb7ed9f5f3a32e43cc5b3553ed0 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/51b35472-39e7-46c0-9beb-edc8484916ed_isrForCrosstalkSources_1126402_60.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/51b35472-39e7-46c0-9beb-edc8484916ed_isrForCrosstalkSources_1126402_60.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash f132d5e321b0a88f2250692f0fb54c94 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943474c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/b33e8f50-7feb-418a-9a93-bcd1406ed685_isrForCrosstalkSources_1126401_38.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/b33e8f50-7feb-418a-9a93-bcd1406ed685_isrForCrosstalkSources_1126401_38.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 5a6d49b26289e474b95d40a9779c7ed8 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/52ebec3b-22bc-401c-a193-345b9523840d_isrForCrosstalkSources_1126401_22.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/52ebec3b-22bc-401c-a193-345b9523840d_isrForCrosstalkSources_1126401_22.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash ab173f99cc76d6d35218cd1c28877d3e parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943476a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b5a0baf6-994d-4c45-8f26-0005cb658cb2_isrForCrosstalkSources_1126402_29.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b5a0baf6-994d-4c45-8f26-0005cb658cb2_isrForCrosstalkSources_1126402_29.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 406f7b595d801e57d636090b44ff1a4c parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/0ef58f1a-3253-4964-aa94-4fe10854e5e7_isrForCrosstalkSources_1126401_13.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/0ef58f1a-3253-4964-aa94-4fe10854e5e7_isrForCrosstalkSources_1126401_13.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 258d25d205c278218e86dd1bfbdc55d9 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/e34a0886-ac38-43aa-9e0b-a42a5e512984_isrForCrosstalkSources_1126402_58.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/e34a0886-ac38-43aa-9e0b-a42a5e512984_isrForCrosstalkSources_1126402_58.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 7595f5bc8f814168f0407e2ae8d7ce35 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943477e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/c47bcfdc-f9bc-4a33-ab8a-3e1c5707ff42_isrForCrosstalkSources_1126400_49.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/c47bcfdc-f9bc-4a33-ab8a-3e1c5707ff42_isrForCrosstalkSources_1126400_49.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 7ae1075bbc289fbf8cc9fadb5ce5e45d parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/3cb0f131-7742-4262-9ad6-1f1fb24929ef_isrForCrosstalkSources_1126400_19.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/3cb0f131-7742-4262-9ad6-1f1fb24929ef_isrForCrosstalkSources_1126400_19.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash eca559c25cebc5b7f3a5300a6c7c1871 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d0943479c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1f460cc7-ba86-431d-a39e-d80f30e91225_isrForCrosstalkSources_1126402_32.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1f460cc7-ba86-431d-a39e-d80f30e91225_isrForCrosstalkSources_1126402_32.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 14a62af8512d14a73cccc6c0c1b2463d parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/4473ed1e-7cab-4fbd-9389-8b3d894d5f1c_isrForCrosstalkSources_1126402_13.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/4473ed1e-7cab-4fbd-9389-8b3d894d5f1c_isrForCrosstalkSources_1126402_13.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 2841f42b8e48c26545b0be4a94bb76e4 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/3bee6e6b-de91-4fe8-9eeb-b5ffcfffd3c8_isrForCrosstalkSources_1126400_39.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/3bee6e6b-de91-4fe8-9eeb-b5ffcfffd3c8_isrForCrosstalkSources_1126400_39.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 651578b77380c2f7dfc641dda3a4f987 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/6c2b1f43-14a8-4a78-a5dc-384629cd6140_isrForCrosstalkSources_1126402_56.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/6c2b1f43-14a8-4a78-a5dc-384629cd6140_isrForCrosstalkSources_1126402_56.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 1050377bccd6bc9ed5eedce8eb610c67 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/552b008e-ab00-4cbd-bccf-b8a4d2499b21_isrForCrosstalkSources_1126402_4.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/552b008e-ab00-4cbd-bccf-b8a4d2499b21_isrForCrosstalkSources_1126402_4.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash a539a9f391c0bad77edd713681a7809e parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2c1a0bbc-67d3-4018-8280-90aaf3395a56_isrForCrosstalkSources_1126402_5.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2c1a0bbc-67d3-4018-8280-90aaf3395a56_isrForCrosstalkSources_1126402_5.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash ee508484ef0b66f5da358a083782f0fd parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/b087a986-78e0-4683-b28e-4272c8b50ae7_isrForCrosstalkSources_1126400_26.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/b087a986-78e0-4683-b28e-4272c8b50ae7_isrForCrosstalkSources_1126400_26.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 0ad73bc758a906b234c0006ac9c2b316 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/6af4c52c-5a8c-493e-b35d-f1ca1d888926_isrForCrosstalkSources_1126400_57.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/6af4c52c-5a8c-493e-b35d-f1ca1d888926_isrForCrosstalkSources_1126400_57.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash ab2cb9d666d9e03ad1ec9c8d20f51dba parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094347e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/005c31fc-3f29-4b37-a304-beef2eab677d_isrForCrosstalkSources_1126400_46.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/005c31fc-3f29-4b37-a304-beef2eab677d_isrForCrosstalkSources_1126400_46.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 20f4cdd0f08f95d8f0d81b0bfa052f46 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d094344680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/59650939-d32b-49de-b4eb-7c89317adc2e_isrForCrosstalkSources_1126401_17.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/59650939-d32b-49de-b4eb-7c89317adc2e_isrForCrosstalkSources_1126401_17.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 49a3b2c5de675507f8a76f56b22f7233 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d980e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/c709dc45-0b57-4ba4-87a7-9ea28cf133e6_isrForCrosstalkSources_1126401_34.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/c709dc45-0b57-4ba4-87a7-9ea28cf133e6_isrForCrosstalkSources_1126401_34.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 141e3ede60aa97183431e891680242bf parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/927666fd-4ed5-4ca6-bccf-76840c702de2_isrForCrosstalkSources_1126400_62.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/927666fd-4ed5-4ca6-bccf-76840c702de2_isrForCrosstalkSources_1126400_62.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash f40eccb088ef66fd206052b18c6469b8 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/6f6140cc-8ede-460b-8eb3-74b1b13170c2_isrForCrosstalkSources_1126401_8.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/6f6140cc-8ede-460b-8eb3-74b1b13170c2_isrForCrosstalkSources_1126401_8.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash d993f036e8e0b81df6eb0dc66b3eb1fe parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d982c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/a7335a34-9e87-4fef-b3fa-690f74b34198_isrForCrosstalkSources_1126400_27.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/a7335a34-9e87-4fef-b3fa-690f74b34198_isrForCrosstalkSources_1126400_27.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 7b46eba864a9ba37c72a5614b0691a2c parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/59f6316a-ecd8-4c7c-8eb2-5104e8642f45_isrForCrosstalkSources_1126400_45.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/59f6316a-ecd8-4c7c-8eb2-5104e8642f45_isrForCrosstalkSources_1126400_45.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash c70f652f2efbb82afe37afcaa6329915 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/fda47b83-5e3a-4f3a-a9b2-02258d3d0b27_isrForCrosstalkSources_1126400_43.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/fda47b83-5e3a-4f3a-a9b2-02258d3d0b27_isrForCrosstalkSources_1126400_43.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 22fd1cdf87802b3c5c570b32518d7ed4 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/472e9911-ac29-422c-8232-0670b0c0a9cd_isrForCrosstalkSources_1126402_40.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/472e9911-ac29-422c-8232-0670b0c0a9cd_isrForCrosstalkSources_1126402_40.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 5afc167e11db650c72cd81ed7d04a6e2 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d984a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fc70301e-a7d8-4bb6-878e-c29605d55350_isrForCrosstalkSources_1126401_37.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fc70301e-a7d8-4bb6-878e-c29605d55350_isrForCrosstalkSources_1126401_37.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 29e44b86ce24fe953a291ed01d81cf0b parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d985e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/1daffbc1-f15c-43ba-a23c-a55ba469d96c_isrForCrosstalkSources_1126401_58.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/1daffbc1-f15c-43ba-a23c-a55ba469d96c_isrForCrosstalkSources_1126401_58.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 447a490f8500ef9d64fc93fdbf4d505b parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/4f1574d0-2aba-4548-9d0a-d11bfdf4c0f4_isrForCrosstalkSources_1126400_7.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/4f1574d0-2aba-4548-9d0a-d11bfdf4c0f4_isrForCrosstalkSources_1126400_7.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 59384848c8e6e7bc562b77aa8c68319e parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/755fde42-84f1-43e0-91da-ebf36e2f882d_isrForCrosstalkSources_1126402_1.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/755fde42-84f1-43e0-91da-ebf36e2f882d_isrForCrosstalkSources_1126402_1.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 862d22860f234e7e3c226f09d94f3d27 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2145be05-1c7c-4181-8339-e956b32c433c_isrForCrosstalkSources_1126402_30.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2145be05-1c7c-4181-8339-e956b32c433c_isrForCrosstalkSources_1126402_30.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 82e5af4456e53816f192ff94ca241555 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d987c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/3cce3bf8-2bb8-4b15-be1a-1afbd4152295_isrForCrosstalkSources_1126400_25.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/3cce3bf8-2bb8-4b15-be1a-1afbd4152295_isrForCrosstalkSources_1126400_25.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash a92762ba67c14032169fd27dad73fa77 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/df822691-c210-45a4-a594-f371d86c09f6_isrForCrosstalkSources_1126402_55.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/df822691-c210-45a4-a594-f371d86c09f6_isrForCrosstalkSources_1126402_55.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash e9dc0ddbba1fa075f8d7990d9c61b29e parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d989a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/d0459571-eec9-41b1-8d05-71c709313f93_isrForCrosstalkSources_1126401_32.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/d0459571-eec9-41b1-8d05-71c709313f93_isrForCrosstalkSources_1126401_32.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 4773517bb43d26d27b825712fa405ca8 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/84729fc8-a5f0-4596-9b60-2cc81a3a7c80_isrForCrosstalkSources_1126402_41.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/84729fc8-a5f0-4596-9b60-2cc81a3a7c80_isrForCrosstalkSources_1126402_41.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash f370fce5ab464ce718c283629be82046 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/fc0fe8c3-d95c-4503-8521-c2dc21db76fc_isrForCrosstalkSources_1126402_14.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/fc0fe8c3-d95c-4503-8521-c2dc21db76fc_isrForCrosstalkSources_1126402_14.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 2245fa7a7624649a7f58316ba8ccf882 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/9abb2060-b0e3-4d1f-8951-e3b403010842_isrForCrosstalkSources_1126401_51.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/9abb2060-b0e3-4d1f-8951-e3b403010842_isrForCrosstalkSources_1126401_51.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 2e965a99d50b1316d01e8c0938745be4 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/e6bc8513-9b25-445c-8663-7aac464bb449_isrForCrosstalkSources_1126402_25.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/e6bc8513-9b25-445c-8663-7aac464bb449_isrForCrosstalkSources_1126402_25.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 5c7e3fde03f1a7d793cefec906c8b256 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/b4418fca-6153-4802-8f1d-fbdf4bdcd728_isrForCrosstalkSources_1126400_11.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/b4418fca-6153-4802-8f1d-fbdf4bdcd728_isrForCrosstalkSources_1126400_11.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash b87a22c478c21272aee4017385767380 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1ae1e40c-a50e-4cec-a682-7aa065e3ffff_isrForCrosstalkSources_1126402_6.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1ae1e40c-a50e-4cec-a682-7aa065e3ffff_isrForCrosstalkSources_1126402_6.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 41e1fd56bdd590692e52b915a06382e3 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cc2d9236-7b44-4271-8395-6d8116d9b3b3_isrForCrosstalkSources_1126401_53.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cc2d9236-7b44-4271-8395-6d8116d9b3b3_isrForCrosstalkSources_1126401_53.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 50a0cf69f05fa3820aa53b95be03c92f parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/19ce2637-995a-42d1-962b-b81ce24d80e0_isrForCrosstalkSources_1126400_32.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/19ce2637-995a-42d1-962b-b81ce24d80e0_isrForCrosstalkSources_1126400_32.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash b9ad97ac0558baa6c96d3b7fc418e564 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2205df6e-2259-41da-96e9-9d7a53414299_isrForCrosstalkSources_1126402_10.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2205df6e-2259-41da-96e9-9d7a53414299_isrForCrosstalkSources_1126402_10.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 81a78d83d313812e4a88b32e9a51668e parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/844a20b7-83f2-445f-ae53-dd3e2d14ac58_isrForCrosstalkSources_1126402_44.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/844a20b7-83f2-445f-ae53-dd3e2d14ac58_isrForCrosstalkSources_1126402_44.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 2e7543320adc12ba4ea71e4156d74639 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/70807706-02e1-4ab2-8637-e5dfeb763fcc_isrForCrosstalkSources_1126400_16.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/70807706-02e1-4ab2-8637-e5dfeb763fcc_isrForCrosstalkSources_1126400_16.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash c98e7b472c0d4b404b07ab618db628a0 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d991c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b6f5b817-52bc-436a-9b4b-d687df5631ff_isrForCrosstalkSources_1126402_18.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b6f5b817-52bc-436a-9b4b-d687df5631ff_isrForCrosstalkSources_1126402_18.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 78f492fbababd667a5d644b7c54bd75f parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/923d90f9-3d4a-42c8-a33a-061246463519_isrForCrosstalkSources_1126401_26.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/923d90f9-3d4a-42c8-a33a-061246463519_isrForCrosstalkSources_1126401_26.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash b5b5200919aa3b34f0b9e1cd05dab40c parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/1ce48928-c332-4bb1-a77b-1ac03ff91df1_isrForCrosstalkSources_1126400_20.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/1ce48928-c332-4bb1-a77b-1ac03ff91df1_isrForCrosstalkSources_1126400_20.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 91a983d0994d48cb6dec99d99b6f4341 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/37220c67-2000-4d9c-94db-d1726f4a6a97_isrForCrosstalkSources_1126402_3.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/37220c67-2000-4d9c-94db-d1726f4a6a97_isrForCrosstalkSources_1126402_3.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 98b30c7e4a7219676d5b1e43b75a5842 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d993a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ef82a546-4cbc-4809-88ff-c020e071b304_isrForCrosstalkSources_1126400_50.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ef82a546-4cbc-4809-88ff-c020e071b304_isrForCrosstalkSources_1126400_50.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 94009771839cb9f2c39212f004d08c3b parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d994e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/e7a946d1-6238-4591-b3ed-cea0a1ef46c5_isrForCrosstalkSources_1126400_13.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/e7a946d1-6238-4591-b3ed-cea0a1ef46c5_isrForCrosstalkSources_1126400_13.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash aaea9a4e80f9264fea9146f84d4cf7bf parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fc8a3100-2e40-4374-8e25-67d63afd43ab_isrForCrosstalkSources_1126401_29.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fc8a3100-2e40-4374-8e25-67d63afd43ab_isrForCrosstalkSources_1126401_29.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 35872975c6743a8d77883624eaf52b56 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/342c0f66-389f-48eb-ae96-456fb60b2225_isrForCrosstalkSources_1126401_15.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/342c0f66-389f-48eb-ae96-456fb60b2225_isrForCrosstalkSources_1126401_15.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 0b61063fdb7bd2a3782b1632de9a558c parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 19:10:43 proc_lsst.multi:146 [INFO] found job 42497 in provider local proc_lsst.multi INFO: found job 42497 in provider local parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/0792ab5c-81a9-466e-9c20-f6c317d02a19_isrForCrosstalkSources_1126401_55.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/0792ab5c-81a9-466e-9c20-f6c317d02a19_isrForCrosstalkSources_1126401_55.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 5b05097be0eab37921bb55100faf6e92 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d996c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b2b90357-7afb-47cc-bbc7-bb22779722fb_isrForCrosstalkSources_1126402_57.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b2b90357-7afb-47cc-bbc7-bb22779722fb_isrForCrosstalkSources_1126402_57.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 985b6f166611f465d7decaeabbe9cc5c parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ef5d09dd-c1f9-430d-b719-2d22f6d0e0e6_isrForCrosstalkSources_1126400_5.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ef5d09dd-c1f9-430d-b719-2d22f6d0e0e6_isrForCrosstalkSources_1126400_5.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 982a5cd35f64f26269a55adf50eb254c parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/6461abee-694a-4cbc-90da-ff85fff08999_isrForCrosstalkSources_1126400_15.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/6461abee-694a-4cbc-90da-ff85fff08999_isrForCrosstalkSources_1126400_15.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash b9e809a8fc052dd15ba70a55884e32da parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/edb1266c-569c-4723-b244-3e812457fc3f_isrForCrosstalkSources_1126400_55.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/edb1266c-569c-4723-b244-3e812457fc3f_isrForCrosstalkSources_1126400_55.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 146160e262a7816b53d85d598ff0acfe parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/4f6758c2-2f9f-466d-b747-15f7f9577a19_isrForCrosstalkSources_1126401_33.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/4f6758c2-2f9f-466d-b747-15f7f9577a19_isrForCrosstalkSources_1126401_33.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash fefd0f11155434c6355dc66452dab852 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ae182cbe-1001-4a4c-93e4-ead31ee59d0f_isrForCrosstalkSources_1126402_16.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ae182cbe-1001-4a4c-93e4-ead31ee59d0f_isrForCrosstalkSources_1126402_16.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 8272e06f8f18d466dfcf0cbc9a3af6c6 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b6ce0811-1d49-4c68-8bbd-f602680f942e_isrForCrosstalkSources_1126402_37.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b6ce0811-1d49-4c68-8bbd-f602680f942e_isrForCrosstalkSources_1126402_37.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 790138d6493372fe5bf50b6d0d555d4b parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ae94db60-47ca-41bd-aa27-306a1cb2cd87_isrForCrosstalkSources_1126400_8.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ae94db60-47ca-41bd-aa27-306a1cb2cd87_isrForCrosstalkSources_1126400_8.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 73a293bb632eee1dbb0b586af88ddefd parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/04f522d0-f94e-456a-ba47-bfd0c05d904e_isrForCrosstalkSources_1126401_36.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/04f522d0-f94e-456a-ba47-bfd0c05d904e_isrForCrosstalkSources_1126401_36.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash df536e6ec0758166935f7c0af2fec2a4 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d98040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ae8a1cbd-0cc0-4fd7-b268-662441d042be_isrForCrosstalkSources_1126400_22.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ae8a1cbd-0cc0-4fd7-b268-662441d042be_isrForCrosstalkSources_1126400_22.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash cffb6e548b5a5592b7744a6dd41abacf parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 151 parsl.dataflow.strategy DEBUG: Executor multi has 151 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 37 more blocks parsl.executors.status_handling INFO: Scaling out by 37 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-07 19:10:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-07 19:10:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 19:10:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/32dfc5f2-be29-4e54-a94a-34861a0508fa_isrForCrosstalkSources_1126401_56.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/32dfc5f2-be29-4e54-a94a-34861a0508fa_isrForCrosstalkSources_1126401_56.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash b1078b9de951d3be6cc27bf594809aa4 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/774425c8-5d53-488f-8a2b-740e6398df15_isrForCrosstalkSources_1126401_19.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/774425c8-5d53-488f-8a2b-740e6398df15_isrForCrosstalkSources_1126401_19.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 63be1c3350163edf347a5cf81b52a580 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d998a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/9a6727f7-69f6-4570-a0bf-9ae87ed3e205_isrForCrosstalkSources_1126400_60.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/9a6727f7-69f6-4570-a0bf-9ae87ed3e205_isrForCrosstalkSources_1126400_60.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash ffcfa7df8e8655944fadebe9b0240650 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d99f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d47bd8af-a847-4f17-96ba-85eb34801ac3_isrForCrosstalkSources_1126400_34.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d47bd8af-a847-4f17-96ba-85eb34801ac3_isrForCrosstalkSources_1126400_34.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 2a5d05b8b5c4c08edf9e0533d50f5270 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/614d0850-7457-4735-9b5d-7fa8c7540bef_isrForCrosstalkSources_1126401_61.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/614d0850-7457-4735-9b5d-7fa8c7540bef_isrForCrosstalkSources_1126401_61.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 4f70d8554f7ef760d2f2bc73335d8c7a parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/44053cd5-1f9d-4fa4-b2ed-5006b7c9a280_isrForCrosstalkSources_1126402_46.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/44053cd5-1f9d-4fa4-b2ed-5006b7c9a280_isrForCrosstalkSources_1126402_46.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 6dc4398efa1e14c9dc0492eaf7254c10 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/5168771f-d258-49e4-beeb-60966dc05c80_isrForCrosstalkSources_1126401_10.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/5168771f-d258-49e4-beeb-60966dc05c80_isrForCrosstalkSources_1126401_10.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 43437d63c73f22863d8c12bbea81c9de parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/fb9d3529-b6a0-4ac4-bdca-d5984eab3559_isrForCrosstalkSources_1126400_53.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/fb9d3529-b6a0-4ac4-bdca-d5984eab3559_isrForCrosstalkSources_1126400_53.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 3a1d7b4768408b28f603f92b129517c8 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d999e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b6b067a9-4f42-420d-af6d-1b67c6836d22_isrForCrosstalkSources_1126402_59.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b6b067a9-4f42-420d-af6d-1b67c6836d22_isrForCrosstalkSources_1126402_59.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash e7e0eef2f583aa9cb4fada4cb32bb1c8 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092dfd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fed14106-11f9-4725-85e8-d1a7165ba454_isrForCrosstalkSources_1126401_20.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fed14106-11f9-4725-85e8-d1a7165ba454_isrForCrosstalkSources_1126401_20.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 397a2f26b84d677de7d33d1b743258e1 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092dfd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/5f20553b-167d-40a6-b590-47215a1d4876_isrForCrosstalkSources_1126402_17.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/5f20553b-167d-40a6-b590-47215a1d4876_isrForCrosstalkSources_1126402_17.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 8b814782ca2fe2b15fa42fd647914b76 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092dfd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/0ae1dea7-f21d-4ee4-b149-1a3466eae49c_isrForCrosstalkSources_1126401_16.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/0ae1dea7-f21d-4ee4-b149-1a3466eae49c_isrForCrosstalkSources_1126401_16.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 2d79051db4618a0c56a26a50d928abaf parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/642287ef-9dd1-4586-b799-9ff4fcfdb777_isrForCrosstalkSources_1126402_12.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/642287ef-9dd1-4586-b799-9ff4fcfdb777_isrForCrosstalkSources_1126402_12.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash ce5d377ff9197b65b9c5172c794152c7 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ea9cd5f9-8aae-4282-9b88-f318dade1974_isrForCrosstalkSources_1126400_10.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ea9cd5f9-8aae-4282-9b88-f318dade1974_isrForCrosstalkSources_1126400_10.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 62c6e99a902ac0a36245cfeb02784640 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/da94ac4c-21f2-4d6a-8dfd-c99805e1e239_isrForCrosstalkSources_1126402_23.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/da94ac4c-21f2-4d6a-8dfd-c99805e1e239_isrForCrosstalkSources_1126402_23.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash a29f19054d64814e678816bc2a4ef6d6 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/fdb28ca2-9e4c-4723-9f2d-749ad411c875_isrForCrosstalkSources_1126400_3.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/fdb28ca2-9e4c-4723-9f2d-749ad411c875_isrForCrosstalkSources_1126400_3.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 07365ea0ef757615dec6fbc6f4b192cb parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/11cd7e02-b9b5-42e1-b8bf-85a20d9dacc0_isrForCrosstalkSources_1126400_56.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/11cd7e02-b9b5-42e1-b8bf-85a20d9dacc0_isrForCrosstalkSources_1126400_56.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 21ada0c1a1315a81c1285551328bcaa6 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/9f1b710e-bd9c-4eba-8394-5b39b55fc5d9_isrForCrosstalkSources_1126401_42.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/9f1b710e-bd9c-4eba-8394-5b39b55fc5d9_isrForCrosstalkSources_1126401_42.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 378e28d2cda991c670de6b231f194522 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/bbec0d13-5a75-4ef5-bf42-ef28ca7295fe_isrForCrosstalkSources_1126401_43.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/bbec0d13-5a75-4ef5-bf42-ef28ca7295fe_isrForCrosstalkSources_1126401_43.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 1a0dd2a34c9b21dd38e0742359ef7bcd parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/61bbfa51-291f-458b-bae2-61bfbcb47f65_isrForCrosstalkSources_1126401_23.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/61bbfa51-291f-458b-bae2-61bfbcb47f65_isrForCrosstalkSources_1126401_23.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash a8bb2dce664889b64bec7b9052466020 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/2145d6cc-768c-45e9-a50e-161bee059886_isrForCrosstalkSources_1126401_62.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/2145d6cc-768c-45e9-a50e-161bee059886_isrForCrosstalkSources_1126401_62.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 72bdf9cc4af1d01b0d246782cb3fc521 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/5f2709a8-f7ab-434c-971f-826d65ea8168_isrForCrosstalkSources_1126400_52.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/5f2709a8-f7ab-434c-971f-826d65ea8168_isrForCrosstalkSources_1126400_52.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 3dde55f1b6194896eac23a1f9c6dc2ee parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/af1b3dfe-c3ac-4920-b473-4fd43c39c2b7_isrForCrosstalkSources_1126401_59.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/af1b3dfe-c3ac-4920-b473-4fd43c39c2b7_isrForCrosstalkSources_1126401_59.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 3a57fc067b06fb4734255f1c2c1a85bc parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/4af5b4f9-8918-4915-8a6e-9f671f73df29_isrForCrosstalkSources_1126401_6.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/4af5b4f9-8918-4915-8a6e-9f671f73df29_isrForCrosstalkSources_1126401_6.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 1e757f4b302bd7e522530d9c6211f75c parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d1e5577e-8365-4dd5-aeb4-d786f6c86f0c_isrForCrosstalkSources_1126400_14.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d1e5577e-8365-4dd5-aeb4-d786f6c86f0c_isrForCrosstalkSources_1126400_14.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash a1f0d84c2385d700097ebf12a732ea74 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ed525685-e49f-40c5-bfe4-ecf2beca4d85_isrForCrosstalkSources_1126402_15.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ed525685-e49f-40c5-bfe4-ecf2beca4d85_isrForCrosstalkSources_1126402_15.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash e2d21c706d25a601d57c6f6c2a00cf96 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/38d28dd1-f59c-4b91-b9eb-8d5cf402e954_isrForCrosstalkSources_1126402_7.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/38d28dd1-f59c-4b91-b9eb-8d5cf402e954_isrForCrosstalkSources_1126402_7.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 4a172db4ebd06c2e81f8327ee42c23f3 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/cd964e71-d7c5-4da2-bb95-61c8da3422a8_isrForCrosstalkSources_1126400_44.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/cd964e71-d7c5-4da2-bb95-61c8da3422a8_isrForCrosstalkSources_1126400_44.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash fb12130369e2b827a6260b57c3674668 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/8737b769-474b-4f8b-b617-a1413d138f71_isrForCrosstalkSources_1126401_11.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/8737b769-474b-4f8b-b617-a1413d138f71_isrForCrosstalkSources_1126401_11.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash a04b3abfc76ad7b03224fa16abfad4b4 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/a9bc6c46-05d4-43b0-b904-608cb5959e49_isrForCrosstalkSources_1126402_61.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/a9bc6c46-05d4-43b0-b904-608cb5959e49_isrForCrosstalkSources_1126402_61.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 6c56a6e12cf0af40fff6725fb9f87ad8 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fd58cee5-9f1e-4f35-8d71-b4b897824410_isrForCrosstalkSources_1126401_4.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fd58cee5-9f1e-4f35-8d71-b4b897824410_isrForCrosstalkSources_1126401_4.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 2647272b2e69650850473d5314c8f969 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14d092d9b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/4666e033-4a7e-4dea-9d7b-01ec11df2498_isrForCrosstalkSources_1126400_6.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/4666e033-4a7e-4dea-9d7b-01ec11df2498_isrForCrosstalkSources_1126400_6.stderr 2024-03-07 19:10:43 proc_lsst.multi:170 [INFO] [multi] job_id 16900724 proc_lsst.multi INFO: [multi] job_id 16900724 2024-03-07 19:10:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900724 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900724 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16900724 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-07 19:10:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-07 19:10:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:44 proc_lsst.multi:170 [INFO] [multi] job_id 16900725 proc_lsst.multi INFO: [multi] job_id 16900725 2024-03-07 19:10:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 19:10:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900725 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900725 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16900725 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-07 19:10:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-07 19:10:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 19:10:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:44 proc_lsst.multi:170 [INFO] [multi] job_id 16900726 proc_lsst.multi INFO: [multi] job_id 16900726 2024-03-07 19:10:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 19:10:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900726 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900726 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16900726 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-07 19:10:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-07 19:10:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 19:10:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:44 proc_lsst.multi:170 [INFO] [multi] job_id 16900727 proc_lsst.multi INFO: [multi] job_id 16900727 2024-03-07 19:10:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 19:10:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900727 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900727 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16900727 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-07 19:10:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-07 19:10:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 19:10:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:45 proc_lsst.multi:170 [INFO] [multi] job_id 16900728 proc_lsst.multi INFO: [multi] job_id 16900728 2024-03-07 19:10:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:10:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900728 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900728 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16900728 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-07 19:10:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-07 19:10:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:45 proc_lsst.multi:170 [INFO] [multi] job_id 16900729 proc_lsst.multi INFO: [multi] job_id 16900729 2024-03-07 19:10:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 19:10:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900729 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900729 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16900729 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-07 19:10:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-07 19:10:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 19:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:45 proc_lsst.multi:170 [INFO] [multi] job_id 16900730 proc_lsst.multi INFO: [multi] job_id 16900730 2024-03-07 19:10:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 19:10:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900730 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900730 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16900730 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-07 19:10:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-07 19:10:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 19:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:46 proc_lsst.multi:170 [INFO] [multi] job_id 16900731 proc_lsst.multi INFO: [multi] job_id 16900731 2024-03-07 19:10:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 19:10:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900731 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900731 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16900731 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-07 19:10:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-07 19:10:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 19:10:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:46 proc_lsst.multi:170 [INFO] [multi] job_id 16900732 proc_lsst.multi INFO: [multi] job_id 16900732 2024-03-07 19:10:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 19:10:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900732 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900732 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16900732 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-07 19:10:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-07 19:10:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 19:10:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:47 proc_lsst.multi:170 [INFO] [multi] job_id 16900733 proc_lsst.multi INFO: [multi] job_id 16900733 2024-03-07 19:10:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 19:10:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900733 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900733 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16900733 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-07 19:10:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-07 19:10:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 19:10:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:47 proc_lsst.multi:170 [INFO] [multi] job_id 16900734 proc_lsst.multi INFO: [multi] job_id 16900734 2024-03-07 19:10:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 19:10:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900734 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900734 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16900734 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-07 19:10:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-07 19:10:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 19:10:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:48 proc_lsst.multi:170 [INFO] [multi] job_id 16900735 proc_lsst.multi INFO: [multi] job_id 16900735 2024-03-07 19:10:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 19:10:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900735 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900735 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16900735 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-07 19:10:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-07 19:10:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 19:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:48 proc_lsst.multi:170 [INFO] [multi] job_id 16900736 proc_lsst.multi INFO: [multi] job_id 16900736 2024-03-07 19:10:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 19:10:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900736 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900736 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16900736 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-07 19:10:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-07 19:10:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 19:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:48 proc_lsst.multi:170 [INFO] [multi] job_id 16900737 proc_lsst.multi INFO: [multi] job_id 16900737 2024-03-07 19:10:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 19:10:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900737 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900737 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16900737 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-07 19:10:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-07 19:10:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 19:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:49 proc_lsst.multi:170 [INFO] [multi] job_id 16900738 proc_lsst.multi INFO: [multi] job_id 16900738 2024-03-07 19:10:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 19:10:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900738 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900738 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16900738 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-07 19:10:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-07 19:10:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 19:10:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:49 proc_lsst.multi:170 [INFO] [multi] job_id 16900739 proc_lsst.multi INFO: [multi] job_id 16900739 2024-03-07 19:10:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 19:10:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900739 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900739 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16900739 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-07 19:10:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-07 19:10:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 19:10:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:49 proc_lsst.multi:170 [INFO] [multi] job_id 16900740 proc_lsst.multi INFO: [multi] job_id 16900740 2024-03-07 19:10:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 19:10:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900740 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900740 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16900740 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-07 19:10:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-07 19:10:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 19:10:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:50 proc_lsst.multi:170 [INFO] [multi] job_id 16900741 proc_lsst.multi INFO: [multi] job_id 16900741 2024-03-07 19:10:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 19:10:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900741 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900741 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16900741 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-07 19:10:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-07 19:10:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 19:10:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:50 proc_lsst.multi:170 [INFO] [multi] job_id 16900742 proc_lsst.multi INFO: [multi] job_id 16900742 2024-03-07 19:10:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 19:10:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900742 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900742 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16900742 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-07 19:10:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-07 19:10:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 19:10:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:51 proc_lsst.multi:170 [INFO] [multi] job_id 16900743 proc_lsst.multi INFO: [multi] job_id 16900743 2024-03-07 19:10:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 19:10:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900743 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900743 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16900743 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-07 19:10:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-07 19:10:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 19:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:51 proc_lsst.multi:170 [INFO] [multi] job_id 16900744 proc_lsst.multi INFO: [multi] job_id 16900744 2024-03-07 19:10:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 19:10:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900744 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900744 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16900744 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-07 19:10:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-07 19:10:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 19:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:52 proc_lsst.multi:170 [INFO] [multi] job_id 16900745 proc_lsst.multi INFO: [multi] job_id 16900745 2024-03-07 19:10:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 19:10:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900745 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900745 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16900745 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-07 19:10:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-07 19:10:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 19:10:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:52 proc_lsst.multi:170 [INFO] [multi] job_id 16900746 proc_lsst.multi INFO: [multi] job_id 16900746 2024-03-07 19:10:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 19:10:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900746 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900746 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16900746 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-07 19:10:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-07 19:10:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 19:10:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:53 proc_lsst.multi:170 [INFO] [multi] job_id 16900747 proc_lsst.multi INFO: [multi] job_id 16900747 2024-03-07 19:10:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 19:10:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900747 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900747 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16900747 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-07 19:10:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-07 19:10:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 19:10:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33f9c18ca9670accd67c3871556cfcde with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/cadb8287-ecfc-486f-80c8-d7c4a3a54657_isrForCrosstalkSources_1126400_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/cadb8287-ecfc-486f-80c8-d7c4a3a54657_isrForCrosstalkSources_1126400_4.stderr 2024-03-07 19:10:53 proc_lsst.multi:170 [INFO] [multi] job_id 16900748 proc_lsst.multi INFO: [multi] job_id 16900748 2024-03-07 19:10:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 19:10:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900748 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900748 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16900748 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-07 19:10:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-07 19:10:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 19:10:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f493a23e49e8b037795659f7beca8e0 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/08d495c4-e1b3-40f8-9a3a-211bdf662c3b_isrForCrosstalkSources_1126400_47.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/08d495c4-e1b3-40f8-9a3a-211bdf662c3b_isrForCrosstalkSources_1126400_47.stderr 2024-03-07 19:10:54 proc_lsst.multi:170 [INFO] [multi] job_id 16900749 proc_lsst.multi INFO: [multi] job_id 16900749 2024-03-07 19:10:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 19:10:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900749 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900749 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16900749 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-07 19:10:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-07 19:10:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 19:10:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:54 proc_lsst.multi:170 [INFO] [multi] job_id 16900750 proc_lsst.multi INFO: [multi] job_id 16900750 2024-03-07 19:10:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 19:10:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900750 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900750 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16900750 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-07 19:10:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-07 19:10:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 19:10:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:55 proc_lsst.multi:170 [INFO] [multi] job_id 16900751 proc_lsst.multi INFO: [multi] job_id 16900751 2024-03-07 19:10:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 19:10:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900751 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900751 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16900751 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-07 19:10:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-07 19:10:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 19:10:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:55 proc_lsst.multi:170 [INFO] [multi] job_id 16900752 proc_lsst.multi INFO: [multi] job_id 16900752 2024-03-07 19:10:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 19:10:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900752 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900752 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16900752 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-07 19:10:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-07 19:10:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 19:10:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:56 proc_lsst.multi:170 [INFO] [multi] job_id 16900753 proc_lsst.multi INFO: [multi] job_id 16900753 2024-03-07 19:10:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:10:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16900753 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16900753 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16900753 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-07 19:10:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-07 19:10:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:10:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 19:10:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:57 proc_lsst.multi:170 [INFO] [multi] job_id 16900754 proc_lsst.multi INFO: [multi] job_id 16900754 2024-03-07 19:10:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:57 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16900754 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16900754 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16900754 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-07 19:10:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-07 19:10:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:10:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:57 proc_lsst.multi:170 [INFO] [multi] job_id 16900755 proc_lsst.multi INFO: [multi] job_id 16900755 2024-03-07 19:10:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 19:10:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:57 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16900755 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16900755 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16900755 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-07 19:10:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-07 19:10:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:10:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 19:10:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:58 proc_lsst.multi:170 [INFO] [multi] job_id 16900756 proc_lsst.multi INFO: [multi] job_id 16900756 2024-03-07 19:10:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 19:10:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:58 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16900756 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16900756 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16900756 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-07 19:10:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-07 19:10:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:10:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 19:10:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 405f3388ad753747ab760c3ed04da969 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/09c9a1be-af8d-4adc-bd6a-1ead23050f1e_isrForCrosstalkSources_1126401_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/09c9a1be-af8d-4adc-bd6a-1ead23050f1e_isrForCrosstalkSources_1126401_25.stderr 2024-03-07 19:10:58 proc_lsst.multi:170 [INFO] [multi] job_id 16900757 proc_lsst.multi INFO: [multi] job_id 16900757 2024-03-07 19:10:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 19:10:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:58 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16900757 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16900757 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16900757 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-07 19:10:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-07 19:10:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:10:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 19:10:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:58 proc_lsst.multi:170 [INFO] [multi] job_id 16900758 proc_lsst.multi INFO: [multi] job_id 16900758 2024-03-07 19:10:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:10:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:58 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16900758 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16900758 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16900758 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-07 19:10:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-07 19:10:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:10:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:10:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 19:10:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:10:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e39b8f0ab777f2cb106b929a304a4d24 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/44bd78df-6e74-42d3-b57e-f86ffe8be357_isrForCrosstalkSources_1126402_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/44bd78df-6e74-42d3-b57e-f86ffe8be357_isrForCrosstalkSources_1126402_11.stderr 2024-03-07 19:10:59 proc_lsst.multi:170 [INFO] [multi] job_id 16900759 proc_lsst.multi INFO: [multi] job_id 16900759 2024-03-07 19:10:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:10:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16900759 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16900759 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16900759 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-07 19:10:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-07 19:10:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:10:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:10:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:10:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:10:59 proc_lsst.multi:170 [INFO] [multi] job_id 16900760 proc_lsst.multi INFO: [multi] job_id 16900760 2024-03-07 19:10:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 19:10:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:10:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16900760 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16900760 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16900760 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22886247474768 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 38, active_tasks = 182 parsl.dataflow.strategy DEBUG: Executor multi has 182 active tasks, 1/37 running/pending blocks, and 12 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 38 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 8 more blocks parsl.executors.status_handling INFO: Scaling out by 8 blocks parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-07 19:10:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-07 19:10:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:10:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:10:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:10:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:10:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:10:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:10:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:10:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:10:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 19:10:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:10:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:11:00 proc_lsst.multi:170 [INFO] [multi] job_id 16900761 proc_lsst.multi INFO: [multi] job_id 16900761 2024-03-07 19:11:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 19:11:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:00 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16900761 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16900761 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16900761 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-07 19:11:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-07 19:11:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:11:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:11:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:11:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:11:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:11:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:11:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:11:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 19:11:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:11:01 proc_lsst.multi:170 [INFO] [multi] job_id 16900762 proc_lsst.multi INFO: [multi] job_id 16900762 2024-03-07 19:11:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 19:11:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:01 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16900762 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16900762 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16900762 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-07 19:11:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-07 19:11:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:11:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:11:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:11:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:11:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:11:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:11:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:11:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 19:11:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:11:01 proc_lsst.multi:170 [INFO] [multi] job_id 16900763 proc_lsst.multi INFO: [multi] job_id 16900763 2024-03-07 19:11:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:11:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:01 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16900763 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16900763 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16900763 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-07 19:11:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-07 19:11:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:11:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:11:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:11:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:11:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:11:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:11:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:11:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:11:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd2a2f8cc80be75b8731b2f8d0f930af with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/cc06a8de-8202-4229-bdc0-34ff2f78d894_isrForCrosstalkSources_1126402_20.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/cc06a8de-8202-4229-bdc0-34ff2f78d894_isrForCrosstalkSources_1126402_20.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 867d19ca01b576f09c65ebe7aa6ee345 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/acfbde09-ba8e-46af-956d-00bebf329e6b_isrForCrosstalkSources_1126401_27.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/acfbde09-ba8e-46af-956d-00bebf329e6b_isrForCrosstalkSources_1126401_27.stderr 2024-03-07 19:11:02 proc_lsst.multi:170 [INFO] [multi] job_id 16900764 proc_lsst.multi INFO: [multi] job_id 16900764 2024-03-07 19:11:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 19:11:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:02 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16900764 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16900764 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16900764 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-07 19:11:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-07 19:11:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:11:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:11:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:11:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:11:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:11:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:11:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:11:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 19:11:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:11:02 proc_lsst.multi:170 [INFO] [multi] job_id 16900765 proc_lsst.multi INFO: [multi] job_id 16900765 2024-03-07 19:11:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 19:11:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:02 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16900765 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16900765 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16900765 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-07 19:11:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-07 19:11:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:11:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:11:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:11:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:11:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:11:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:11:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:11:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 19:11:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bae5548b30035a365b371d847f065cc with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/6348e86a-37b2-4bb1-b06f-e7858f7e1c71_isrForCrosstalkSources_1126401_9.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/6348e86a-37b2-4bb1-b06f-e7858f7e1c71_isrForCrosstalkSources_1126401_9.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e3fa13735476dfbd801ef5a4c024f33 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/090ae0d2-9469-4976-a6a6-eb9307b3c459_isrForCrosstalkSources_1126401_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/090ae0d2-9469-4976-a6a6-eb9307b3c459_isrForCrosstalkSources_1126401_48.stderr 2024-03-07 19:11:03 proc_lsst.multi:170 [INFO] [multi] job_id 16900766 proc_lsst.multi INFO: [multi] job_id 16900766 2024-03-07 19:11:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 19:11:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:03 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16900766 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16900766 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16900766 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-07 19:11:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-07 19:11:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:11:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:11:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:11:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:11:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:11:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:11:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:11:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 19:11:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:11:03 proc_lsst.multi:170 [INFO] [multi] job_id 16900767 proc_lsst.multi INFO: [multi] job_id 16900767 2024-03-07 19:11:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 19:11:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:03 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16900767 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16900767 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16900767 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-07 19:11:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-07 19:11:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 19:11:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 19:11:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 19:11:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 19:11:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 19:11:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 19:11:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 19:11:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 19:11:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 19:11:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54232 --result_port=54836 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 19:11:04 proc_lsst.multi:170 [INFO] [multi] job_id 16900768 proc_lsst.multi INFO: [multi] job_id 16900768 2024-03-07 19:11:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 19:11:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 19:11:04 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16900768 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16900768 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16900768 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22886247474768 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 178 parsl.dataflow.strategy DEBUG: Executor multi has 178 active tasks, 1/45 running/pending blocks, and 15 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22886247474768 parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78cd12fea61e16f2440aeae11c408377 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/0ff7ad2a-0d19-4e8c-9c06-8d77556f79c4_isrForCrosstalkSources_1126402_33.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/0ff7ad2a-0d19-4e8c-9c06-8d77556f79c4_isrForCrosstalkSources_1126402_33.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0841d32994dafe263c68894a6d7b7d1c with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/8744cf61-89ef-4362-ae82-f5453c5716f7_isrForCrosstalkSources_1126400_40.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/8744cf61-89ef-4362-ae82-f5453c5716f7_isrForCrosstalkSources_1126400_40.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81fe8caf5a82b1c6eb063788ecd7a053 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/384226e6-f30f-436f-b623-4fc900aa84a5_isrForCrosstalkSources_1126401_39.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/384226e6-f30f-436f-b623-4fc900aa84a5_isrForCrosstalkSources_1126401_39.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 175 parsl.dataflow.strategy DEBUG: Executor multi has 175 active tasks, 1/45 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22886247474768 parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26ea9e1089af403877836876d9384497 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/27717ee2-1dff-4ff6-bde9-bdd6f33fdc78_isrForCrosstalkSources_1126400_61.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/27717ee2-1dff-4ff6-bde9-bdd6f33fdc78_isrForCrosstalkSources_1126400_61.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry beed196ce6b81d4896db0fa3bab99502 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/09e88ef5-1859-4991-b073-0938209d738c_isrForCrosstalkSources_1126401_40.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/09e88ef5-1859-4991-b073-0938209d738c_isrForCrosstalkSources_1126401_40.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e34425344caedbe4f2e9be7b6860d87f with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d30b5e8c-4d0b-4eb8-a9b5-c959733ce81e_isrForCrosstalkSources_1126400_21.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d30b5e8c-4d0b-4eb8-a9b5-c959733ce81e_isrForCrosstalkSources_1126400_21.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eb3505095230a835bdd15861c7915e0 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/a88865d8-dfca-4eb7-b899-3a7da5fe443d_isrForCrosstalkSources_1126401_45.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/a88865d8-dfca-4eb7-b899-3a7da5fe443d_isrForCrosstalkSources_1126401_45.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f596777bd971dec1697c38f49aafcb8 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/6eb989e1-afe0-4019-b123-88b6975b6340_isrForCrosstalkSources_1126400_17.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/6eb989e1-afe0-4019-b123-88b6975b6340_isrForCrosstalkSources_1126400_17.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f152782a4cb9e3c6f1ed82ef230bcfb with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/950b505d-1fd1-409b-aa8c-98632c9d591f_isrForCrosstalkSources_1126400_54.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/950b505d-1fd1-409b-aa8c-98632c9d591f_isrForCrosstalkSources_1126400_54.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b355a8d8ce938af4aab0f7bba0ca6b6 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/79c1dc82-6267-4316-912c-8bd4897c8169_isrForCrosstalkSources_1126401_28.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/79c1dc82-6267-4316-912c-8bd4897c8169_isrForCrosstalkSources_1126401_28.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9d97a2e7dd9e28cc0f0655876f6545d with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/f7b883d5-8741-447e-b13a-d5512dafabef_isrForCrosstalkSources_1126402_43.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/f7b883d5-8741-447e-b13a-d5512dafabef_isrForCrosstalkSources_1126402_43.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34b47d43dff31fb85ad58b9170d8781f with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/a69fe013-b834-491f-97f9-b93c0e9f1d19_isrForCrosstalkSources_1126402_62.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/a69fe013-b834-491f-97f9-b93c0e9f1d19_isrForCrosstalkSources_1126402_62.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60286cc8b02241cb0f7c3e54ee327364 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/af60ada3-fa34-4744-8f10-b2f0936b45f1_isrForCrosstalkSources_1126402_24.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/af60ada3-fa34-4744-8f10-b2f0936b45f1_isrForCrosstalkSources_1126402_24.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0109625180f9332174a05cc3a3018083 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d3888e6f-28e3-4360-b224-d128efd8030a_isrForCrosstalkSources_1126400_23.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d3888e6f-28e3-4360-b224-d128efd8030a_isrForCrosstalkSources_1126400_23.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f07466577330841093198cbcd56670b with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/2ac9160e-cda0-4afc-a590-1ac0f739a258_isrForCrosstalkSources_1126400_41.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/2ac9160e-cda0-4afc-a590-1ac0f739a258_isrForCrosstalkSources_1126400_41.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc34405ee8caa4d385a5d56d1aedfa7f with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1d024acc-5a13-434c-8e93-1613be78f5dd_isrForCrosstalkSources_1126402_51.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1d024acc-5a13-434c-8e93-1613be78f5dd_isrForCrosstalkSources_1126402_51.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c63c8ea0ac42539045ba2664e5e31d6 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cdae455e-d342-4022-959a-e8b03d06230c_isrForCrosstalkSources_1126401_3.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cdae455e-d342-4022-959a-e8b03d06230c_isrForCrosstalkSources_1126401_3.stderr 2024-03-07 19:11:14 proc_lsst.multi:146 [INFO] found job 42497 in provider local proc_lsst.multi INFO: found job 42497 in provider local 2024-03-07 19:11:15 proc_lsst.multi:146 [INFO] found job 16900724 in provider astro proc_lsst.multi INFO: found job 16900724 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6a03521238276269936066d760097da with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/6ab5d426-bcac-4d5a-acb8-b11d21c16b6b_isrForCrosstalkSources_1126401_41.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/6ab5d426-bcac-4d5a-acb8-b11d21c16b6b_isrForCrosstalkSources_1126401_41.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab162b52f552fad6a1394e708df4913b with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/cf1d847e-96f0-44c3-9593-6b5c9e25960a_isrForCrosstalkSources_1126402_45.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/cf1d847e-96f0-44c3-9593-6b5c9e25960a_isrForCrosstalkSources_1126402_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:15 proc_lsst.multi:146 [INFO] found job 16900725 in provider astro proc_lsst.multi INFO: found job 16900725 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 400e4d7fdff95c7859cc450db9cbfc10 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/5aa4efca-7897-44fc-b9d5-b28c6235fc5a_isrForCrosstalkSources_1126401_18.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/5aa4efca-7897-44fc-b9d5-b28c6235fc5a_isrForCrosstalkSources_1126401_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:16 proc_lsst.multi:146 [INFO] found job 16900726 in provider astro proc_lsst.multi INFO: found job 16900726 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:17 proc_lsst.multi:146 [INFO] found job 16900727 in provider astro proc_lsst.multi INFO: found job 16900727 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:17 proc_lsst.multi:146 [INFO] found job 16900728 in provider astro proc_lsst.multi INFO: found job 16900728 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99b261e1d18b9ea445687344e7ccd32c with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/e607ac5c-b8a0-42f4-8ca4-2c2d7b8d0489_isrForCrosstalkSources_1126400_12.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/e607ac5c-b8a0-42f4-8ca4-2c2d7b8d0489_isrForCrosstalkSources_1126400_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:18 proc_lsst.multi:146 [INFO] found job 16900729 in provider astro proc_lsst.multi INFO: found job 16900729 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac659dd2ea07afa8845feb9feb798871 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/a653ab59-12cb-4167-b8f4-f20474ab7b01_isrForCrosstalkSources_1126401_30.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/a653ab59-12cb-4167-b8f4-f20474ab7b01_isrForCrosstalkSources_1126401_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:19 proc_lsst.multi:146 [INFO] found job 16900730 in provider astro proc_lsst.multi INFO: found job 16900730 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a687db940405a7216d7f2fca39d18400 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cbf23c54-1eea-4fc0-9677-56f07ae1563e_isrForCrosstalkSources_1126401_57.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cbf23c54-1eea-4fc0-9677-56f07ae1563e_isrForCrosstalkSources_1126401_57.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7fa8aa6c8496036ecc7e128adf33a0b with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/dfdbbcbd-f674-44f6-973d-aca2f9b5ec8c_isrForCrosstalkSources_1126401_54.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/dfdbbcbd-f674-44f6-973d-aca2f9b5ec8c_isrForCrosstalkSources_1126401_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:19 proc_lsst.multi:146 [INFO] found job 16900731 in provider astro proc_lsst.multi INFO: found job 16900731 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f9fdf4fe12c318a066e0addb38f43c0 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/68b91aca-64bb-40bb-8269-28cfd1ead066_isrForCrosstalkSources_1126400_35.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/68b91aca-64bb-40bb-8269-28cfd1ead066_isrForCrosstalkSources_1126400_35.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b98df085da5226c18d847f2f6d0916a4 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ac5e93d0-a697-4af4-95e9-5cc6f3952891_isrForCrosstalkSources_1126400_1.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ac5e93d0-a697-4af4-95e9-5cc6f3952891_isrForCrosstalkSources_1126400_1.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8c6c7ef9931448cabde4356d68e679e with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/447b5c4f-8131-4ff9-8714-83f55b89a6a2_isrForCrosstalkSources_1126401_46.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/447b5c4f-8131-4ff9-8714-83f55b89a6a2_isrForCrosstalkSources_1126401_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:20 proc_lsst.multi:146 [INFO] found job 16900732 in provider astro proc_lsst.multi INFO: found job 16900732 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42e6d2303e1d11fd90e9cd930244ea18 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ed6e664a-79e4-4292-98af-449194a2719f_isrForCrosstalkSources_1126400_51.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ed6e664a-79e4-4292-98af-449194a2719f_isrForCrosstalkSources_1126400_51.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f88cc46f29e9037ba9bcc92dc9c379db with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1f797ac3-94e1-42de-8952-1266e4aa7776_isrForCrosstalkSources_1126402_2.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1f797ac3-94e1-42de-8952-1266e4aa7776_isrForCrosstalkSources_1126402_2.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 635ae86deeff6e9952b8844716c324a4 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/31845923-9ebe-41c9-b0ad-f42fd26dd003_isrForCrosstalkSources_1126402_36.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/31845923-9ebe-41c9-b0ad-f42fd26dd003_isrForCrosstalkSources_1126402_36.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8ec515438886f8753ba6eff575a93da with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/5a5a2f1f-6cc8-4276-b208-9c7f24f402a1_isrForCrosstalkSources_1126402_42.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/5a5a2f1f-6cc8-4276-b208-9c7f24f402a1_isrForCrosstalkSources_1126402_42.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1709d152ad0fddacd0f3db1bf319adfe with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/eacadb65-b819-4e4b-b57b-328d50aaf7dc_isrForCrosstalkSources_1126400_42.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/eacadb65-b819-4e4b-b57b-328d50aaf7dc_isrForCrosstalkSources_1126400_42.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 431daee726fe304f8d31e4d04399449d with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/22d7be72-01d4-4276-a9b8-c1a7a954aeb2_isrForCrosstalkSources_1126402_48.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/22d7be72-01d4-4276-a9b8-c1a7a954aeb2_isrForCrosstalkSources_1126402_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:21 proc_lsst.multi:146 [INFO] found job 16900733 in provider astro proc_lsst.multi INFO: found job 16900733 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fe8998b577f534416b939d72b54ccec with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/006d64bf-4dc0-4a2d-8e36-0fbc6f7fe4fa_isrForCrosstalkSources_1126402_54.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/006d64bf-4dc0-4a2d-8e36-0fbc6f7fe4fa_isrForCrosstalkSources_1126402_54.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 263ac898bf28a2a8884d2eb350c6c5d7 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/3923fb1b-5b19-483e-a3d8-c2d90bf3b071_isrForCrosstalkSources_1126401_44.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/3923fb1b-5b19-483e-a3d8-c2d90bf3b071_isrForCrosstalkSources_1126401_44.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06bbbd186046b6e535d8ea074d0b00ff with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/05ee01ea-45ff-4c8a-b2fd-81bb0f419a30_isrForCrosstalkSources_1126402_9.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/05ee01ea-45ff-4c8a-b2fd-81bb0f419a30_isrForCrosstalkSources_1126402_9.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d934f75066a1b895de36e049e56d35cb with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/93ea9278-80ab-4e23-a8f1-897ff0716b47_isrForCrosstalkSources_1126400_9.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/93ea9278-80ab-4e23-a8f1-897ff0716b47_isrForCrosstalkSources_1126400_9.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1121a35e14bd2fc7df8954a93be4b785 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1c092bcf-031b-49b4-869e-34ec3536d0de_isrForCrosstalkSources_1126402_27.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1c092bcf-031b-49b4-869e-34ec3536d0de_isrForCrosstalkSources_1126402_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:21 proc_lsst.multi:146 [INFO] found job 16900734 in provider astro proc_lsst.multi INFO: found job 16900734 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:22 proc_lsst.multi:146 [INFO] found job 16900735 in provider astro proc_lsst.multi INFO: found job 16900735 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03176aad0238c064332d7f046f5a9172 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fb727ed7-ca16-4831-b859-8397adc4c2c2_isrForCrosstalkSources_1126401_21.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fb727ed7-ca16-4831-b859-8397adc4c2c2_isrForCrosstalkSources_1126401_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:23 proc_lsst.multi:146 [INFO] found job 16900736 in provider astro proc_lsst.multi INFO: found job 16900736 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efdd0ac20d6a98a82dd3c17b5f2ee7d6 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/5e4f06f5-a364-4d78-b125-9188d4d644b3_isrForCrosstalkSources_1126401_7.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/5e4f06f5-a364-4d78-b125-9188d4d644b3_isrForCrosstalkSources_1126401_7.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a1458fd40b6c1c0329bbcf57bd736e4 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/3d916b70-2b81-4813-9fa8-0f950b25c3e8_isrForCrosstalkSources_1126402_19.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/3d916b70-2b81-4813-9fa8-0f950b25c3e8_isrForCrosstalkSources_1126402_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:23 proc_lsst.multi:146 [INFO] found job 16900737 in provider astro proc_lsst.multi INFO: found job 16900737 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc7eba193d5f42085f5763b6ca8cb019 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/f423c78f-5423-46de-a377-9df0e8b47d65_isrForCrosstalkSources_1126401_12.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/f423c78f-5423-46de-a377-9df0e8b47d65_isrForCrosstalkSources_1126401_12.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ea5b06c1a086927228bff5d64570efb with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/099868cc-903d-4b38-b52d-4d1a209cee86_isrForCrosstalkSources_1126400_28.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/099868cc-903d-4b38-b52d-4d1a209cee86_isrForCrosstalkSources_1126400_28.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d7638c5e5f36523c1f060f1cbb1eeab with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/528f2f35-d740-4be0-875a-a205ef019930_isrForCrosstalkSources_1126402_52.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/528f2f35-d740-4be0-875a-a205ef019930_isrForCrosstalkSources_1126402_52.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 202e05bbe8d03846cfaadd7333f9a2a2 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/48e58e4f-32b1-4ccb-8225-81587f9c8de4_isrForCrosstalkSources_1126401_60.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/48e58e4f-32b1-4ccb-8225-81587f9c8de4_isrForCrosstalkSources_1126401_60.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d77058671d01c9674ef5c2ab726139f8 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/5fbc53da-00dd-4ef4-8271-ea147c7d0692_isrForCrosstalkSources_1126400_37.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/5fbc53da-00dd-4ef4-8271-ea147c7d0692_isrForCrosstalkSources_1126400_37.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc867c18245ec0b33bc3cdc0a3215f3c with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/4256510f-e43d-4421-9257-230f8cd64f36_isrForCrosstalkSources_1126402_53.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/4256510f-e43d-4421-9257-230f8cd64f36_isrForCrosstalkSources_1126402_53.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fd17f6bb6c011fa7f1a1dc31bc9c95d with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/796a034b-2b2f-49f3-900f-c1899c74a0e1_isrForCrosstalkSources_1126401_49.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/796a034b-2b2f-49f3-900f-c1899c74a0e1_isrForCrosstalkSources_1126401_49.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f1565d40eb43d452ee16946b68d8784 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/c01bc846-60ef-4a47-8e04-b08902c4da2c_isrForCrosstalkSources_1126400_48.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/c01bc846-60ef-4a47-8e04-b08902c4da2c_isrForCrosstalkSources_1126400_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:27 proc_lsst.multi:146 [INFO] found job 16900738 in provider astro proc_lsst.multi INFO: found job 16900738 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6acc1e216abfac6d040b1494e9a1eec3 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/dd011805-93ea-440e-849e-7fbb2b7c11f5_isrForCrosstalkSources_1126400_31.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/dd011805-93ea-440e-849e-7fbb2b7c11f5_isrForCrosstalkSources_1126400_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:28 proc_lsst.multi:146 [INFO] found job 16900739 in provider astro proc_lsst.multi INFO: found job 16900739 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c2471da8558d39a8a54573c737ef45a with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/034c916f-5818-4b0c-ac78-a4a0d9cf8445_isrForCrosstalkSources_1126401_31.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/034c916f-5818-4b0c-ac78-a4a0d9cf8445_isrForCrosstalkSources_1126401_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:28 proc_lsst.multi:146 [INFO] found job 16900740 in provider astro proc_lsst.multi INFO: found job 16900740 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae09f11a068a884b7a76f85b95f55e9e with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/698362b7-9080-4138-8555-62f1f98236eb_isrForCrosstalkSources_1126400_2.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/698362b7-9080-4138-8555-62f1f98236eb_isrForCrosstalkSources_1126400_2.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a24857a4e5bead499a1345a977ed569 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/f3008da6-1da6-44ce-9c8e-befe01bcca12_isrForCrosstalkSources_1126400_29.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/f3008da6-1da6-44ce-9c8e-befe01bcca12_isrForCrosstalkSources_1126400_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:29 proc_lsst.multi:146 [INFO] found job 16900741 in provider astro proc_lsst.multi INFO: found job 16900741 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ac1c5311ca7ee5a050b677d1f21f56d with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/824cbe35-8c21-4f40-a881-7adbbd087e82_isrForCrosstalkSources_1126400_36.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/824cbe35-8c21-4f40-a881-7adbbd087e82_isrForCrosstalkSources_1126400_36.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b35d458bba01c58b4077f6fd0d5b2018 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/16116073-2467-49d4-9f35-33c0dae62c19_isrForCrosstalkSources_1126402_31.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/16116073-2467-49d4-9f35-33c0dae62c19_isrForCrosstalkSources_1126402_31.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c20407558b21c5c2ea48915c4c0cf15a with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/621b695d-2f29-48c3-bef5-9726653b4073_isrForCrosstalkSources_1126401_35.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/621b695d-2f29-48c3-bef5-9726653b4073_isrForCrosstalkSources_1126401_35.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0afcaccf46054405244e50f48592e57 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/61f20329-fdb8-4620-9163-696e12430594_isrForCrosstalkSources_1126401_5.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/61f20329-fdb8-4620-9163-696e12430594_isrForCrosstalkSources_1126401_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:30 proc_lsst.multi:146 [INFO] found job 16900742 in provider astro proc_lsst.multi INFO: found job 16900742 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6e89493b8421763b0f27cab0cc45087 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cd1bbf6d-7c7c-48ef-abf7-936bd44a6d0e_isrForCrosstalkSources_1126401_47.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cd1bbf6d-7c7c-48ef-abf7-936bd44a6d0e_isrForCrosstalkSources_1126401_47.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 424c3ceaad67bc444aea935d95c70da7 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/7cc72bf7-9cdc-4574-802c-3018c179dd09_isrForCrosstalkSources_1126400_30.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/7cc72bf7-9cdc-4574-802c-3018c179dd09_isrForCrosstalkSources_1126400_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:30 proc_lsst.multi:146 [INFO] found job 16900743 in provider astro proc_lsst.multi INFO: found job 16900743 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ac6ff2911cad48812c6ee2568628b4a with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/754bdbb2-7ed0-4e2f-9496-39e593bcf33c_isrForCrosstalkSources_1126402_34.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/754bdbb2-7ed0-4e2f-9496-39e593bcf33c_isrForCrosstalkSources_1126402_34.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fe0a50225ce8a1d13d32365bc8bdb09 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/3b3c3dc3-ed50-4a07-bae6-c90e5463a659_isrForCrosstalkSources_1126402_8.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/3b3c3dc3-ed50-4a07-bae6-c90e5463a659_isrForCrosstalkSources_1126402_8.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d1e18833af064b67c6e9a62a1dfe4e0 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/caac1eee-a6b6-48b4-b276-392ce6969afa_isrForCrosstalkSources_1126400_59.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/caac1eee-a6b6-48b4-b276-392ce6969afa_isrForCrosstalkSources_1126400_59.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5744b12f5658d2485e10ab4fa241b2f3 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b64f9fdc-c36d-41c0-a2a4-3ae0fcd3b67f_isrForCrosstalkSources_1126402_35.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b64f9fdc-c36d-41c0-a2a4-3ae0fcd3b67f_isrForCrosstalkSources_1126402_35.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96392a46d7deba90bef26922686fbd4e with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/3a9b3b4a-f9ae-4c94-949b-e3a97d45fffc_isrForCrosstalkSources_1126401_52.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/3a9b3b4a-f9ae-4c94-949b-e3a97d45fffc_isrForCrosstalkSources_1126401_52.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e31407fdf9f8eaccdce38abb4eeba08e with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/8eae20ba-7256-4748-94df-cc1184025650_isrForCrosstalkSources_1126402_50.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/8eae20ba-7256-4748-94df-cc1184025650_isrForCrosstalkSources_1126402_50.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 232b419f3f3fc0cecd50559d606b2538 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ccb883c1-356a-4815-a3df-4a4884e029e0_isrForCrosstalkSources_1126402_22.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ccb883c1-356a-4815-a3df-4a4884e029e0_isrForCrosstalkSources_1126402_22.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59501d9047be181e9dafe52bb7b74727 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/e826bc89-0019-44db-b0a7-f21b12c06ba1_isrForCrosstalkSources_1126401_2.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/e826bc89-0019-44db-b0a7-f21b12c06ba1_isrForCrosstalkSources_1126401_2.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0f7795858ef697ad33c5c60496c9853 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/38a4f24d-813d-4d85-8292-d51d0daf6e93_isrForCrosstalkSources_1126402_47.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/38a4f24d-813d-4d85-8292-d51d0daf6e93_isrForCrosstalkSources_1126402_47.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a5bb3788537a4c70e673a92d49adcd4 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/81c6e095-c13f-42fe-b86d-77e0ed1e88a3_isrForCrosstalkSources_1126401_1.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/81c6e095-c13f-42fe-b86d-77e0ed1e88a3_isrForCrosstalkSources_1126401_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:34 proc_lsst.multi:146 [INFO] found job 16900744 in provider astro proc_lsst.multi INFO: found job 16900744 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:34 proc_lsst.multi:146 [INFO] found job 16900745 in provider astro proc_lsst.multi INFO: found job 16900745 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b1effa071b3f70127cd4bed6942e4cb with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2f48c7c8-4544-4ec0-9b56-5844e6ab10db_isrForCrosstalkSources_1126402_21.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2f48c7c8-4544-4ec0-9b56-5844e6ab10db_isrForCrosstalkSources_1126402_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:35 proc_lsst.multi:146 [INFO] found job 16900746 in provider astro proc_lsst.multi INFO: found job 16900746 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5854a8dd130aa4dc1f92df0e67b355d with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/c8b123ec-91be-4200-8997-1d0d1cd544b8_isrForCrosstalkSources_1126401_14.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/c8b123ec-91be-4200-8997-1d0d1cd544b8_isrForCrosstalkSources_1126401_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:35 proc_lsst.multi:146 [INFO] found job 16900747 in provider astro proc_lsst.multi INFO: found job 16900747 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb988793fa8c05c3be218ff4337b8ea0 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/0d7f0945-f031-4b5e-b3a1-92ffe37973e7_isrForCrosstalkSources_1126400_33.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/0d7f0945-f031-4b5e-b3a1-92ffe37973e7_isrForCrosstalkSources_1126400_33.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cda25871978fab921d4ad27177a61897 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/eb1fc4cf-bdef-4bdd-b1ad-754904d5f3cd_isrForCrosstalkSources_1126401_24.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/eb1fc4cf-bdef-4bdd-b1ad-754904d5f3cd_isrForCrosstalkSources_1126401_24.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3af98fc769fab058a95e70738ea85e7 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/7fe262a3-96d9-47ef-bac2-a9c165f12956_isrForCrosstalkSources_1126400_38.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/7fe262a3-96d9-47ef-bac2-a9c165f12956_isrForCrosstalkSources_1126400_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:36 proc_lsst.multi:146 [INFO] found job 16900748 in provider astro proc_lsst.multi INFO: found job 16900748 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f9902006a5d8f14d709a1a58041984f with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ae6bb2c5-d585-4a15-81f9-9b34c5273e0a_isrForCrosstalkSources_1126402_49.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ae6bb2c5-d585-4a15-81f9-9b34c5273e0a_isrForCrosstalkSources_1126402_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:37 proc_lsst.multi:146 [INFO] found job 16900749 in provider astro proc_lsst.multi INFO: found job 16900749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6c1273c26eada7179d16bbca2cca814 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/d1a68a75-3908-4b8b-af82-74de954bfe67_isrForCrosstalkSources_1126401_50.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/d1a68a75-3908-4b8b-af82-74de954bfe67_isrForCrosstalkSources_1126401_50.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd2bfca7d18acf6b47a8c77be40cb986 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/aa79c2b0-f917-426d-8125-4e3d393a0162_isrForCrosstalkSources_1126402_26.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/aa79c2b0-f917-426d-8125-4e3d393a0162_isrForCrosstalkSources_1126402_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:37 proc_lsst.multi:146 [INFO] found job 16900750 in provider astro proc_lsst.multi INFO: found job 16900750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8c190034c5fb8e600728d1f51076ec4 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/8b315f22-bc98-4177-a07f-bf4a5de4591f_isrForCrosstalkSources_1126400_24.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/8b315f22-bc98-4177-a07f-bf4a5de4591f_isrForCrosstalkSources_1126400_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:38 proc_lsst.multi:146 [INFO] found job 16900751 in provider astro proc_lsst.multi INFO: found job 16900751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2173de0aef4857bbafb3a59d2427676d with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/42e48eff-19aa-484a-b1e5-0dbebc9aaa83_isrForCrosstalkSources_1126402_39.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/42e48eff-19aa-484a-b1e5-0dbebc9aaa83_isrForCrosstalkSources_1126402_39.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d238909265c9955da7d7dc645df24848 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/413c8a5c-3732-46d9-bf59-adc79c415004_isrForCrosstalkSources_1126402_38.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/413c8a5c-3732-46d9-bf59-adc79c415004_isrForCrosstalkSources_1126402_38.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab173f99cc76d6d35218cd1c28877d3e with result from task 94 parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:39 proc_lsst.multi:146 [INFO] found job 16900752 in provider astro proc_lsst.multi INFO: found job 16900752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b5a0baf6-994d-4c45-8f26-0005cb658cb2_isrForCrosstalkSources_1126402_29.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b5a0baf6-994d-4c45-8f26-0005cb658cb2_isrForCrosstalkSources_1126402_29.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f132d5e321b0a88f2250692f0fb54c94 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/b33e8f50-7feb-418a-9a93-bcd1406ed685_isrForCrosstalkSources_1126401_38.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/b33e8f50-7feb-418a-9a93-bcd1406ed685_isrForCrosstalkSources_1126401_38.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cd121f486d689c27abb5a9a7e9e2cee with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/b8d5a844-7dd8-4deb-b548-0f73ac63954b_isrForCrosstalkSources_1126400_18.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/b8d5a844-7dd8-4deb-b548-0f73ac63954b_isrForCrosstalkSources_1126400_18.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 406f7b595d801e57d636090b44ff1a4c with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/0ef58f1a-3253-4964-aa94-4fe10854e5e7_isrForCrosstalkSources_1126401_13.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/0ef58f1a-3253-4964-aa94-4fe10854e5e7_isrForCrosstalkSources_1126401_13.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 258d25d205c278218e86dd1bfbdc55d9 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/e34a0886-ac38-43aa-9e0b-a42a5e512984_isrForCrosstalkSources_1126402_58.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/e34a0886-ac38-43aa-9e0b-a42a5e512984_isrForCrosstalkSources_1126402_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:39 proc_lsst.multi:146 [INFO] found job 16900753 in provider astro proc_lsst.multi INFO: found job 16900753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbf3386c10d637c85fdd7fcea15cdd20 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d6971f35-ca0a-4322-a575-d803854781b7_isrForCrosstalkSources_1126400_58.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d6971f35-ca0a-4322-a575-d803854781b7_isrForCrosstalkSources_1126400_58.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16310f6b52f62a6154d33d35412066f2 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/83f5f68d-9fb2-4811-a6df-0c519379a04f_isrForCrosstalkSources_1126402_28.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/83f5f68d-9fb2-4811-a6df-0c519379a04f_isrForCrosstalkSources_1126402_28.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 651578b77380c2f7dfc641dda3a4f987 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/6c2b1f43-14a8-4a78-a5dc-384629cd6140_isrForCrosstalkSources_1126402_56.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/6c2b1f43-14a8-4a78-a5dc-384629cd6140_isrForCrosstalkSources_1126402_56.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d788cb7ed9f5f3a32e43cc5b3553ed0 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/51b35472-39e7-46c0-9beb-edc8484916ed_isrForCrosstalkSources_1126402_60.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/51b35472-39e7-46c0-9beb-edc8484916ed_isrForCrosstalkSources_1126402_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900734 PD 16900735 PD 16900736 PD 16900737 PD 16900738 PD 16900739 PD 16900740 PD 16900741 PD 16900742 PD 16900743 PD 16900744 PD 16900745 PD 16900746 PD 16900747 PD 16900748 PD 16900749 PD 16900750 PD 16900751 PD 16900752 PD 16900753 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:40 proc_lsst.multi:146 [INFO] found job 16900754 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900754 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900754,16900755,16900756,16900757,16900758' parsl.providers.slurm.slurm DEBUG: squeue returned 16900755 R 16900756 R 16900757 R 16900758 R 16900754 R parsl.providers.slurm.slurm DEBUG: Updating job 16900755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900754 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:41 proc_lsst.multi:146 [INFO] found job 16900755 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900755 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900754,16900755,16900756,16900757,16900758' parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a6d49b26289e474b95d40a9779c7ed8 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/52ebec3b-22bc-401c-a193-345b9523840d_isrForCrosstalkSources_1126401_22.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/52ebec3b-22bc-401c-a193-345b9523840d_isrForCrosstalkSources_1126401_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900755 R 16900756 R 16900757 R 16900758 R 16900754 R parsl.providers.slurm.slurm DEBUG: Updating job 16900755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900754 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:41 proc_lsst.multi:146 [INFO] found job 16900756 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900756 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900754,16900755,16900756,16900757,16900758' parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eca559c25cebc5b7f3a5300a6c7c1871 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1f460cc7-ba86-431d-a39e-d80f30e91225_isrForCrosstalkSources_1126402_32.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1f460cc7-ba86-431d-a39e-d80f30e91225_isrForCrosstalkSources_1126402_32.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ae1075bbc289fbf8cc9fadb5ce5e45d with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/3cb0f131-7742-4262-9ad6-1f1fb24929ef_isrForCrosstalkSources_1126400_19.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/3cb0f131-7742-4262-9ad6-1f1fb24929ef_isrForCrosstalkSources_1126400_19.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14a62af8512d14a73cccc6c0c1b2463d with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/4473ed1e-7cab-4fbd-9389-8b3d894d5f1c_isrForCrosstalkSources_1126402_13.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/4473ed1e-7cab-4fbd-9389-8b3d894d5f1c_isrForCrosstalkSources_1126402_13.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2841f42b8e48c26545b0be4a94bb76e4 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/3bee6e6b-de91-4fe8-9eeb-b5ffcfffd3c8_isrForCrosstalkSources_1126400_39.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/3bee6e6b-de91-4fe8-9eeb-b5ffcfffd3c8_isrForCrosstalkSources_1126400_39.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7595f5bc8f814168f0407e2ae8d7ce35 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/c47bcfdc-f9bc-4a33-ab8a-3e1c5707ff42_isrForCrosstalkSources_1126400_49.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/c47bcfdc-f9bc-4a33-ab8a-3e1c5707ff42_isrForCrosstalkSources_1126400_49.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1050377bccd6bc9ed5eedce8eb610c67 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/552b008e-ab00-4cbd-bccf-b8a4d2499b21_isrForCrosstalkSources_1126402_4.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/552b008e-ab00-4cbd-bccf-b8a4d2499b21_isrForCrosstalkSources_1126402_4.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a539a9f391c0bad77edd713681a7809e with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2c1a0bbc-67d3-4018-8280-90aaf3395a56_isrForCrosstalkSources_1126402_5.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2c1a0bbc-67d3-4018-8280-90aaf3395a56_isrForCrosstalkSources_1126402_5.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee508484ef0b66f5da358a083782f0fd with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/b087a986-78e0-4683-b28e-4272c8b50ae7_isrForCrosstalkSources_1126400_26.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/b087a986-78e0-4683-b28e-4272c8b50ae7_isrForCrosstalkSources_1126400_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900755 R 16900756 R 16900757 R 16900758 R 16900754 R parsl.providers.slurm.slurm DEBUG: Updating job 16900755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900754 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:43 proc_lsst.multi:146 [INFO] found job 16900757 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900757 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900754,16900755,16900756,16900757,16900758' parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab2cb9d666d9e03ad1ec9c8d20f51dba with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/005c31fc-3f29-4b37-a304-beef2eab677d_isrForCrosstalkSources_1126400_46.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/005c31fc-3f29-4b37-a304-beef2eab677d_isrForCrosstalkSources_1126400_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900755 R 16900756 R 16900757 R 16900758 R 16900754 R parsl.providers.slurm.slurm DEBUG: Updating job 16900755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900754 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:44 proc_lsst.multi:146 [INFO] found job 16900758 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900758 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900754,16900755,16900756,16900757,16900758' parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20f4cdd0f08f95d8f0d81b0bfa052f46 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/59650939-d32b-49de-b4eb-7c89317adc2e_isrForCrosstalkSources_1126401_17.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/59650939-d32b-49de-b4eb-7c89317adc2e_isrForCrosstalkSources_1126401_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900755 R 16900756 R 16900757 R 16900758 R 16900754 R parsl.providers.slurm.slurm DEBUG: Updating job 16900755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900754 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:44 proc_lsst.multi:146 [INFO] found job 16900759 in provider escience_a40 proc_lsst.multi INFO: found job 16900759 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 141e3ede60aa97183431e891680242bf with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/927666fd-4ed5-4ca6-bccf-76840c702de2_isrForCrosstalkSources_1126400_62.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/927666fd-4ed5-4ca6-bccf-76840c702de2_isrForCrosstalkSources_1126400_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:44 proc_lsst.multi:146 [INFO] found job 16900760 in provider escience_a40 proc_lsst.multi INFO: found job 16900760 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f40eccb088ef66fd206052b18c6469b8 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/6f6140cc-8ede-460b-8eb3-74b1b13170c2_isrForCrosstalkSources_1126401_8.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/6f6140cc-8ede-460b-8eb3-74b1b13170c2_isrForCrosstalkSources_1126401_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:45 proc_lsst.multi:146 [INFO] found job 16900761 in provider escience_a40 proc_lsst.multi INFO: found job 16900761 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:45 proc_lsst.multi:146 [INFO] found job 16900762 in provider escience_a40 proc_lsst.multi INFO: found job 16900762 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49a3b2c5de675507f8a76f56b22f7233 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/c709dc45-0b57-4ba4-87a7-9ea28cf133e6_isrForCrosstalkSources_1126401_34.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/c709dc45-0b57-4ba4-87a7-9ea28cf133e6_isrForCrosstalkSources_1126401_34.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d993f036e8e0b81df6eb0dc66b3eb1fe with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/a7335a34-9e87-4fef-b3fa-690f74b34198_isrForCrosstalkSources_1126400_27.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/a7335a34-9e87-4fef-b3fa-690f74b34198_isrForCrosstalkSources_1126400_27.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22fd1cdf87802b3c5c570b32518d7ed4 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/472e9911-ac29-422c-8232-0670b0c0a9cd_isrForCrosstalkSources_1126402_40.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/472e9911-ac29-422c-8232-0670b0c0a9cd_isrForCrosstalkSources_1126402_40.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ad73bc758a906b234c0006ac9c2b316 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/6af4c52c-5a8c-493e-b35d-f1ca1d888926_isrForCrosstalkSources_1126400_57.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/6af4c52c-5a8c-493e-b35d-f1ca1d888926_isrForCrosstalkSources_1126400_57.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 447a490f8500ef9d64fc93fdbf4d505b with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/4f1574d0-2aba-4548-9d0a-d11bfdf4c0f4_isrForCrosstalkSources_1126400_7.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/4f1574d0-2aba-4548-9d0a-d11bfdf4c0f4_isrForCrosstalkSources_1126400_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:48 proc_lsst.multi:146 [INFO] found job 16900763 in provider escience_a40 proc_lsst.multi INFO: found job 16900763 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59384848c8e6e7bc562b77aa8c68319e with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/755fde42-84f1-43e0-91da-ebf36e2f882d_isrForCrosstalkSources_1126402_1.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/755fde42-84f1-43e0-91da-ebf36e2f882d_isrForCrosstalkSources_1126402_1.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a92762ba67c14032169fd27dad73fa77 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/df822691-c210-45a4-a594-f371d86c09f6_isrForCrosstalkSources_1126402_55.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/df822691-c210-45a4-a594-f371d86c09f6_isrForCrosstalkSources_1126402_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:49 proc_lsst.multi:146 [INFO] found job 16900764 in provider escience_a40 proc_lsst.multi INFO: found job 16900764 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82e5af4456e53816f192ff94ca241555 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/3cce3bf8-2bb8-4b15-be1a-1afbd4152295_isrForCrosstalkSources_1126400_25.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/3cce3bf8-2bb8-4b15-be1a-1afbd4152295_isrForCrosstalkSources_1126400_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:50 proc_lsst.multi:146 [INFO] found job 16900765 in provider escience_a40 proc_lsst.multi INFO: found job 16900765 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 862d22860f234e7e3c226f09d94f3d27 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2145be05-1c7c-4181-8339-e956b32c433c_isrForCrosstalkSources_1126402_30.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2145be05-1c7c-4181-8339-e956b32c433c_isrForCrosstalkSources_1126402_30.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9dc0ddbba1fa075f8d7990d9c61b29e with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/d0459571-eec9-41b1-8d05-71c709313f93_isrForCrosstalkSources_1126401_32.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/d0459571-eec9-41b1-8d05-71c709313f93_isrForCrosstalkSources_1126401_32.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b46eba864a9ba37c72a5614b0691a2c with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/59f6316a-ecd8-4c7c-8eb2-5104e8642f45_isrForCrosstalkSources_1126400_45.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/59f6316a-ecd8-4c7c-8eb2-5104e8642f45_isrForCrosstalkSources_1126400_45.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5afc167e11db650c72cd81ed7d04a6e2 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fc70301e-a7d8-4bb6-878e-c29605d55350_isrForCrosstalkSources_1126401_37.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fc70301e-a7d8-4bb6-878e-c29605d55350_isrForCrosstalkSources_1126401_37.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50a0cf69f05fa3820aa53b95be03c92f with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/19ce2637-995a-42d1-962b-b81ce24d80e0_isrForCrosstalkSources_1126400_32.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/19ce2637-995a-42d1-962b-b81ce24d80e0_isrForCrosstalkSources_1126400_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:50 proc_lsst.multi:146 [INFO] found job 16900766 in provider escience_a40 proc_lsst.multi INFO: found job 16900766 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c70f652f2efbb82afe37afcaa6329915 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/fda47b83-5e3a-4f3a-a9b2-02258d3d0b27_isrForCrosstalkSources_1126400_43.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/fda47b83-5e3a-4f3a-a9b2-02258d3d0b27_isrForCrosstalkSources_1126400_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:51 proc_lsst.multi:146 [INFO] found job 16900767 in provider escience_a40 proc_lsst.multi INFO: found job 16900767 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f370fce5ab464ce718c283629be82046 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/fc0fe8c3-d95c-4503-8521-c2dc21db76fc_isrForCrosstalkSources_1126402_14.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/fc0fe8c3-d95c-4503-8521-c2dc21db76fc_isrForCrosstalkSources_1126402_14.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e965a99d50b1316d01e8c0938745be4 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/e6bc8513-9b25-445c-8663-7aac464bb449_isrForCrosstalkSources_1126402_25.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/e6bc8513-9b25-445c-8663-7aac464bb449_isrForCrosstalkSources_1126402_25.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81a78d83d313812e4a88b32e9a51668e with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/844a20b7-83f2-445f-ae53-dd3e2d14ac58_isrForCrosstalkSources_1126402_44.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/844a20b7-83f2-445f-ae53-dd3e2d14ac58_isrForCrosstalkSources_1126402_44.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2245fa7a7624649a7f58316ba8ccf882 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/9abb2060-b0e3-4d1f-8951-e3b403010842_isrForCrosstalkSources_1126401_51.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/9abb2060-b0e3-4d1f-8951-e3b403010842_isrForCrosstalkSources_1126401_51.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c7e3fde03f1a7d793cefec906c8b256 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/b4418fca-6153-4802-8f1d-fbdf4bdcd728_isrForCrosstalkSources_1126400_11.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/b4418fca-6153-4802-8f1d-fbdf4bdcd728_isrForCrosstalkSources_1126400_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:52 proc_lsst.multi:146 [INFO] found job 16900768 in provider escience_a40 proc_lsst.multi INFO: found job 16900768 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29e44b86ce24fe953a291ed01d81cf0b with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/1daffbc1-f15c-43ba-a23c-a55ba469d96c_isrForCrosstalkSources_1126401_58.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/1daffbc1-f15c-43ba-a23c-a55ba469d96c_isrForCrosstalkSources_1126401_58.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c98e7b472c0d4b404b07ab618db628a0 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b6f5b817-52bc-436a-9b4b-d687df5631ff_isrForCrosstalkSources_1126402_18.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b6f5b817-52bc-436a-9b4b-d687df5631ff_isrForCrosstalkSources_1126402_18.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b87a22c478c21272aee4017385767380 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1ae1e40c-a50e-4cec-a682-7aa065e3ffff_isrForCrosstalkSources_1126402_6.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/1ae1e40c-a50e-4cec-a682-7aa065e3ffff_isrForCrosstalkSources_1126402_6.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e7543320adc12ba4ea71e4156d74639 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/70807706-02e1-4ab2-8637-e5dfeb763fcc_isrForCrosstalkSources_1126400_16.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/70807706-02e1-4ab2-8637-e5dfeb763fcc_isrForCrosstalkSources_1126400_16.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41e1fd56bdd590692e52b915a06382e3 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cc2d9236-7b44-4271-8395-6d8116d9b3b3_isrForCrosstalkSources_1126401_53.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/cc2d9236-7b44-4271-8395-6d8116d9b3b3_isrForCrosstalkSources_1126401_53.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4773517bb43d26d27b825712fa405ca8 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/84729fc8-a5f0-4596-9b60-2cc81a3a7c80_isrForCrosstalkSources_1126402_41.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/84729fc8-a5f0-4596-9b60-2cc81a3a7c80_isrForCrosstalkSources_1126402_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 51 parsl.dataflow.strategy DEBUG: Executor multi has 51 active tasks, 26/20 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22886247474768 2024-03-07 19:11:52 proc_lsst.multi:146 [INFO] found job 42497 in provider local proc_lsst.multi INFO: found job 42497 in provider local 2024-03-07 19:11:52 proc_lsst.multi:146 [INFO] found job 16900724 in provider astro proc_lsst.multi INFO: found job 16900724 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78f492fbababd667a5d644b7c54bd75f with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/923d90f9-3d4a-42c8-a33a-061246463519_isrForCrosstalkSources_1126401_26.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/923d90f9-3d4a-42c8-a33a-061246463519_isrForCrosstalkSources_1126401_26.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9ad97ac0558baa6c96d3b7fc418e564 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2205df6e-2259-41da-96e9-9d7a53414299_isrForCrosstalkSources_1126402_10.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/2205df6e-2259-41da-96e9-9d7a53414299_isrForCrosstalkSources_1126402_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:53 proc_lsst.multi:146 [INFO] found job 16900725 in provider astro proc_lsst.multi INFO: found job 16900725 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91a983d0994d48cb6dec99d99b6f4341 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/37220c67-2000-4d9c-94db-d1726f4a6a97_isrForCrosstalkSources_1126402_3.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/37220c67-2000-4d9c-94db-d1726f4a6a97_isrForCrosstalkSources_1126402_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:54 proc_lsst.multi:146 [INFO] found job 16900726 in provider astro proc_lsst.multi INFO: found job 16900726 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98b30c7e4a7219676d5b1e43b75a5842 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ef82a546-4cbc-4809-88ff-c020e071b304_isrForCrosstalkSources_1126400_50.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ef82a546-4cbc-4809-88ff-c020e071b304_isrForCrosstalkSources_1126400_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:54 proc_lsst.multi:146 [INFO] found job 16900727 in provider astro proc_lsst.multi INFO: found job 16900727 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:55 proc_lsst.multi:146 [INFO] found job 16900728 in provider astro proc_lsst.multi INFO: found job 16900728 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaea9a4e80f9264fea9146f84d4cf7bf with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fc8a3100-2e40-4374-8e25-67d63afd43ab_isrForCrosstalkSources_1126401_29.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fc8a3100-2e40-4374-8e25-67d63afd43ab_isrForCrosstalkSources_1126401_29.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35872975c6743a8d77883624eaf52b56 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/342c0f66-389f-48eb-ae96-456fb60b2225_isrForCrosstalkSources_1126401_15.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/342c0f66-389f-48eb-ae96-456fb60b2225_isrForCrosstalkSources_1126401_15.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5b5200919aa3b34f0b9e1cd05dab40c with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/1ce48928-c332-4bb1-a77b-1ac03ff91df1_isrForCrosstalkSources_1126400_20.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/1ce48928-c332-4bb1-a77b-1ac03ff91df1_isrForCrosstalkSources_1126400_20.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94009771839cb9f2c39212f004d08c3b with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/e7a946d1-6238-4591-b3ed-cea0a1ef46c5_isrForCrosstalkSources_1126400_13.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/e7a946d1-6238-4591-b3ed-cea0a1ef46c5_isrForCrosstalkSources_1126400_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:56 proc_lsst.multi:146 [INFO] found job 16900729 in provider astro proc_lsst.multi INFO: found job 16900729 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:56 proc_lsst.multi:146 [INFO] found job 16900730 in provider astro proc_lsst.multi INFO: found job 16900730 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b61063fdb7bd2a3782b1632de9a558c with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/0792ab5c-81a9-466e-9c20-f6c317d02a19_isrForCrosstalkSources_1126401_55.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/0792ab5c-81a9-466e-9c20-f6c317d02a19_isrForCrosstalkSources_1126401_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:57 proc_lsst.multi:146 [INFO] found job 16900731 in provider astro proc_lsst.multi INFO: found job 16900731 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:58 proc_lsst.multi:146 [INFO] found job 16900732 in provider astro proc_lsst.multi INFO: found job 16900732 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:58 proc_lsst.multi:146 [INFO] found job 16900733 in provider astro proc_lsst.multi INFO: found job 16900733 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fefd0f11155434c6355dc66452dab852 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ae182cbe-1001-4a4c-93e4-ead31ee59d0f_isrForCrosstalkSources_1126402_16.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ae182cbe-1001-4a4c-93e4-ead31ee59d0f_isrForCrosstalkSources_1126402_16.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df536e6ec0758166935f7c0af2fec2a4 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ae8a1cbd-0cc0-4fd7-b268-662441d042be_isrForCrosstalkSources_1126400_22.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ae8a1cbd-0cc0-4fd7-b268-662441d042be_isrForCrosstalkSources_1126400_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:11:59 proc_lsst.multi:146 [INFO] found job 16900734 in provider astro proc_lsst.multi INFO: found job 16900734 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 146160e262a7816b53d85d598ff0acfe with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/4f6758c2-2f9f-466d-b747-15f7f9577a19_isrForCrosstalkSources_1126401_33.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/4f6758c2-2f9f-466d-b747-15f7f9577a19_isrForCrosstalkSources_1126401_33.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b05097be0eab37921bb55100faf6e92 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b2b90357-7afb-47cc-bbc7-bb22779722fb_isrForCrosstalkSources_1126402_57.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b2b90357-7afb-47cc-bbc7-bb22779722fb_isrForCrosstalkSources_1126402_57.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 982a5cd35f64f26269a55adf50eb254c with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/6461abee-694a-4cbc-90da-ff85fff08999_isrForCrosstalkSources_1126400_15.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/6461abee-694a-4cbc-90da-ff85fff08999_isrForCrosstalkSources_1126400_15.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 985b6f166611f465d7decaeabbe9cc5c with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ef5d09dd-c1f9-430d-b719-2d22f6d0e0e6_isrForCrosstalkSources_1126400_5.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ef5d09dd-c1f9-430d-b719-2d22f6d0e0e6_isrForCrosstalkSources_1126400_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:00 proc_lsst.multi:146 [INFO] found job 16900735 in provider astro proc_lsst.multi INFO: found job 16900735 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffcfa7df8e8655944fadebe9b0240650 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d47bd8af-a847-4f17-96ba-85eb34801ac3_isrForCrosstalkSources_1126400_34.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d47bd8af-a847-4f17-96ba-85eb34801ac3_isrForCrosstalkSources_1126400_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:00 proc_lsst.multi:146 [INFO] found job 16900736 in provider astro proc_lsst.multi INFO: found job 16900736 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a1d7b4768408b28f603f92b129517c8 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b6b067a9-4f42-420d-af6d-1b67c6836d22_isrForCrosstalkSources_1126402_59.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b6b067a9-4f42-420d-af6d-1b67c6836d22_isrForCrosstalkSources_1126402_59.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 790138d6493372fe5bf50b6d0d555d4b with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ae94db60-47ca-41bd-aa27-306a1cb2cd87_isrForCrosstalkSources_1126400_8.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ae94db60-47ca-41bd-aa27-306a1cb2cd87_isrForCrosstalkSources_1126400_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:01 proc_lsst.multi:146 [INFO] found job 16900737 in provider astro proc_lsst.multi INFO: found job 16900737 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43437d63c73f22863d8c12bbea81c9de with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/fb9d3529-b6a0-4ac4-bdca-d5984eab3559_isrForCrosstalkSources_1126400_53.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/fb9d3529-b6a0-4ac4-bdca-d5984eab3559_isrForCrosstalkSources_1126400_53.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1078b9de951d3be6cc27bf594809aa4 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/774425c8-5d53-488f-8a2b-740e6398df15_isrForCrosstalkSources_1126401_19.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/774425c8-5d53-488f-8a2b-740e6398df15_isrForCrosstalkSources_1126401_19.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cffb6e548b5a5592b7744a6dd41abacf with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/32dfc5f2-be29-4e54-a94a-34861a0508fa_isrForCrosstalkSources_1126401_56.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/32dfc5f2-be29-4e54-a94a-34861a0508fa_isrForCrosstalkSources_1126401_56.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f70d8554f7ef760d2f2bc73335d8c7a with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/44053cd5-1f9d-4fa4-b2ed-5006b7c9a280_isrForCrosstalkSources_1126402_46.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/44053cd5-1f9d-4fa4-b2ed-5006b7c9a280_isrForCrosstalkSources_1126402_46.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73a293bb632eee1dbb0b586af88ddefd with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/04f522d0-f94e-456a-ba47-bfd0c05d904e_isrForCrosstalkSources_1126401_36.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/04f522d0-f94e-456a-ba47-bfd0c05d904e_isrForCrosstalkSources_1126401_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:01 proc_lsst.multi:146 [INFO] found job 16900738 in provider astro proc_lsst.multi INFO: found job 16900738 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7e0eef2f583aa9cb4fada4cb32bb1c8 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fed14106-11f9-4725-85e8-d1a7165ba454_isrForCrosstalkSources_1126401_20.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fed14106-11f9-4725-85e8-d1a7165ba454_isrForCrosstalkSources_1126401_20.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 397a2f26b84d677de7d33d1b743258e1 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/5f20553b-167d-40a6-b590-47215a1d4876_isrForCrosstalkSources_1126402_17.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/5f20553b-167d-40a6-b590-47215a1d4876_isrForCrosstalkSources_1126402_17.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a5d05b8b5c4c08edf9e0533d50f5270 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/614d0850-7457-4735-9b5d-7fa8c7540bef_isrForCrosstalkSources_1126401_61.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/614d0850-7457-4735-9b5d-7fa8c7540bef_isrForCrosstalkSources_1126401_61.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dc4398efa1e14c9dc0492eaf7254c10 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/5168771f-d258-49e4-beeb-60966dc05c80_isrForCrosstalkSources_1126401_10.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/5168771f-d258-49e4-beeb-60966dc05c80_isrForCrosstalkSources_1126401_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:02 proc_lsst.multi:146 [INFO] found job 16900739 in provider astro proc_lsst.multi INFO: found job 16900739 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b814782ca2fe2b15fa42fd647914b76 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/0ae1dea7-f21d-4ee4-b149-1a3466eae49c_isrForCrosstalkSources_1126401_16.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/0ae1dea7-f21d-4ee4-b149-1a3466eae49c_isrForCrosstalkSources_1126401_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:03 proc_lsst.multi:146 [INFO] found job 16900740 in provider astro proc_lsst.multi INFO: found job 16900740 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9e809a8fc052dd15ba70a55884e32da with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/edb1266c-569c-4723-b244-3e812457fc3f_isrForCrosstalkSources_1126400_55.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/edb1266c-569c-4723-b244-3e812457fc3f_isrForCrosstalkSources_1126400_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:03 proc_lsst.multi:146 [INFO] found job 16900741 in provider astro proc_lsst.multi INFO: found job 16900741 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce5d377ff9197b65b9c5172c794152c7 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ea9cd5f9-8aae-4282-9b88-f318dade1974_isrForCrosstalkSources_1126400_10.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/ea9cd5f9-8aae-4282-9b88-f318dade1974_isrForCrosstalkSources_1126400_10.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d79051db4618a0c56a26a50d928abaf with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/642287ef-9dd1-4586-b799-9ff4fcfdb777_isrForCrosstalkSources_1126402_12.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/642287ef-9dd1-4586-b799-9ff4fcfdb777_isrForCrosstalkSources_1126402_12.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8272e06f8f18d466dfcf0cbc9a3af6c6 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b6ce0811-1d49-4c68-8bbd-f602680f942e_isrForCrosstalkSources_1126402_37.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/b6ce0811-1d49-4c68-8bbd-f602680f942e_isrForCrosstalkSources_1126402_37.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63be1c3350163edf347a5cf81b52a580 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/9a6727f7-69f6-4570-a0bf-9ae87ed3e205_isrForCrosstalkSources_1126400_60.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/9a6727f7-69f6-4570-a0bf-9ae87ed3e205_isrForCrosstalkSources_1126400_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:04 proc_lsst.multi:146 [INFO] found job 16900742 in provider astro proc_lsst.multi INFO: found job 16900742 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62c6e99a902ac0a36245cfeb02784640 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/da94ac4c-21f2-4d6a-8dfd-c99805e1e239_isrForCrosstalkSources_1126402_23.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/da94ac4c-21f2-4d6a-8dfd-c99805e1e239_isrForCrosstalkSources_1126402_23.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a29f19054d64814e678816bc2a4ef6d6 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/fdb28ca2-9e4c-4723-9f2d-749ad411c875_isrForCrosstalkSources_1126400_3.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/fdb28ca2-9e4c-4723-9f2d-749ad411c875_isrForCrosstalkSources_1126400_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:04 proc_lsst.multi:146 [INFO] found job 16900743 in provider astro proc_lsst.multi INFO: found job 16900743 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07365ea0ef757615dec6fbc6f4b192cb with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/11cd7e02-b9b5-42e1-b8bf-85a20d9dacc0_isrForCrosstalkSources_1126400_56.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/11cd7e02-b9b5-42e1-b8bf-85a20d9dacc0_isrForCrosstalkSources_1126400_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:05 proc_lsst.multi:146 [INFO] found job 16900744 in provider astro proc_lsst.multi INFO: found job 16900744 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:06 proc_lsst.multi:146 [INFO] found job 16900745 in provider astro proc_lsst.multi INFO: found job 16900745 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:06 proc_lsst.multi:146 [INFO] found job 16900746 in provider astro proc_lsst.multi INFO: found job 16900746 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:06 proc_lsst.multi:146 [INFO] found job 16900747 in provider astro proc_lsst.multi INFO: found job 16900747 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21ada0c1a1315a81c1285551328bcaa6 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/9f1b710e-bd9c-4eba-8394-5b39b55fc5d9_isrForCrosstalkSources_1126401_42.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/9f1b710e-bd9c-4eba-8394-5b39b55fc5d9_isrForCrosstalkSources_1126401_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:07 proc_lsst.multi:146 [INFO] found job 16900748 in provider astro proc_lsst.multi INFO: found job 16900748 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 378e28d2cda991c670de6b231f194522 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/bbec0d13-5a75-4ef5-bf42-ef28ca7295fe_isrForCrosstalkSources_1126401_43.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/bbec0d13-5a75-4ef5-bf42-ef28ca7295fe_isrForCrosstalkSources_1126401_43.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a57fc067b06fb4734255f1c2c1a85bc with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/4af5b4f9-8918-4915-8a6e-9f671f73df29_isrForCrosstalkSources_1126401_6.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/4af5b4f9-8918-4915-8a6e-9f671f73df29_isrForCrosstalkSources_1126401_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:08 proc_lsst.multi:146 [INFO] found job 16900749 in provider astro proc_lsst.multi INFO: found job 16900749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a0dd2a34c9b21dd38e0742359ef7bcd with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/61bbfa51-291f-458b-bae2-61bfbcb47f65_isrForCrosstalkSources_1126401_23.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/61bbfa51-291f-458b-bae2-61bfbcb47f65_isrForCrosstalkSources_1126401_23.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e757f4b302bd7e522530d9c6211f75c with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d1e5577e-8365-4dd5-aeb4-d786f6c86f0c_isrForCrosstalkSources_1126400_14.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/d1e5577e-8365-4dd5-aeb4-d786f6c86f0c_isrForCrosstalkSources_1126400_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:08 proc_lsst.multi:146 [INFO] found job 16900750 in provider astro proc_lsst.multi INFO: found job 16900750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8bb2dce664889b64bec7b9052466020 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/2145d6cc-768c-45e9-a50e-161bee059886_isrForCrosstalkSources_1126401_62.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/2145d6cc-768c-45e9-a50e-161bee059886_isrForCrosstalkSources_1126401_62.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dde55f1b6194896eac23a1f9c6dc2ee with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/af1b3dfe-c3ac-4920-b473-4fd43c39c2b7_isrForCrosstalkSources_1126401_59.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/af1b3dfe-c3ac-4920-b473-4fd43c39c2b7_isrForCrosstalkSources_1126401_59.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72bdf9cc4af1d01b0d246782cb3fc521 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/5f2709a8-f7ab-434c-971f-826d65ea8168_isrForCrosstalkSources_1126400_52.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/5f2709a8-f7ab-434c-971f-826d65ea8168_isrForCrosstalkSources_1126400_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:09 proc_lsst.multi:146 [INFO] found job 16900751 in provider astro proc_lsst.multi INFO: found job 16900751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2d21c706d25a601d57c6f6c2a00cf96 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/38d28dd1-f59c-4b91-b9eb-8d5cf402e954_isrForCrosstalkSources_1126402_7.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/38d28dd1-f59c-4b91-b9eb-8d5cf402e954_isrForCrosstalkSources_1126402_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:09 proc_lsst.multi:146 [INFO] found job 16900752 in provider astro proc_lsst.multi INFO: found job 16900752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:10 proc_lsst.multi:146 [INFO] found job 16900753 in provider astro proc_lsst.multi INFO: found job 16900753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900724,16900725,16900726,16900727,16900728,16900729,16900730,16900731,16900732,16900733,16900734,16900735,16900736,16900737,16900738,16900739,16900740,16900741,16900742,16900743,16900744,16900745,16900746,16900747,16900748,16900749,16900750,16900751,16900752,16900753' parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2647272b2e69650850473d5314c8f969 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/4666e033-4a7e-4dea-9d7b-01ec11df2498_isrForCrosstalkSources_1126400_6.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/4666e033-4a7e-4dea-9d7b-01ec11df2498_isrForCrosstalkSources_1126400_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900753 PD 16900752 PD 16900751 PD 16900750 PD 16900749 PD 16900748 PD 16900747 PD 16900746 PD 16900745 PD 16900744 PD 16900743 PD 16900742 PD 16900741 PD 16900740 PD 16900739 PD 16900738 PD 16900737 PD 16900736 PD 16900735 PD 16900734 PD 16900726 R 16900727 R 16900728 R 16900729 R 16900730 R 16900731 R 16900732 R 16900733 R 16900724 R 16900725 R parsl.providers.slurm.slurm DEBUG: Updating job 16900753 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900752 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900751 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900750 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900749 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900748 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900747 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900746 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900745 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900744 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900743 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900742 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900741 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900740 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900739 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900738 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900737 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900736 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900735 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900734 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16900726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900725 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:10 proc_lsst.multi:146 [INFO] found job 16900754 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900754 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900754,16900755,16900756,16900757,16900758' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a04b3abfc76ad7b03224fa16abfad4b4 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/a9bc6c46-05d4-43b0-b904-608cb5959e49_isrForCrosstalkSources_1126402_61.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/a9bc6c46-05d4-43b0-b904-608cb5959e49_isrForCrosstalkSources_1126402_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900755 R 16900756 R 16900757 R 16900758 R 16900754 R parsl.providers.slurm.slurm DEBUG: Updating job 16900755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900754 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:10 proc_lsst.multi:146 [INFO] found job 16900755 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900755 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900754,16900755,16900756,16900757,16900758' parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb12130369e2b827a6260b57c3674668 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/8737b769-474b-4f8b-b617-a1413d138f71_isrForCrosstalkSources_1126401_11.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/8737b769-474b-4f8b-b617-a1413d138f71_isrForCrosstalkSources_1126401_11.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1f0d84c2385d700097ebf12a732ea74 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ed525685-e49f-40c5-bfe4-ecf2beca4d85_isrForCrosstalkSources_1126402_15.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126402/ed525685-e49f-40c5-bfe4-ecf2beca4d85_isrForCrosstalkSources_1126402_15.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c56a6e12cf0af40fff6725fb9f87ad8 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fd58cee5-9f1e-4f35-8d71-b4b897824410_isrForCrosstalkSources_1126401_4.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126401/fd58cee5-9f1e-4f35-8d71-b4b897824410_isrForCrosstalkSources_1126401_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16900755 R 16900756 R 16900757 R 16900758 R 16900754 R parsl.providers.slurm.slurm DEBUG: Updating job 16900755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900754 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:10 proc_lsst.multi:146 [INFO] found job 16900756 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900756 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900754,16900755,16900756,16900757,16900758' parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a172db4ebd06c2e81f8327ee42c23f3 with result from task 181 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 186 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/cd964e71-d7c5-4da2-bb95-61c8da3422a8_isrForCrosstalkSources_1126400_44.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/submit/DEEP/20220827/B1l/science#step0/20240307T191019Z/logs/isrForCrosstalkSources/1126400/cd964e71-d7c5-4da2-bb95-61c8da3422a8_isrForCrosstalkSources_1126400_44.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 16900755 R 16900756 R 16900757 R 16900758 R 16900754 R parsl.providers.slurm.slurm DEBUG: Updating job 16900755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900754 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:11 proc_lsst.multi:146 [INFO] found job 16900757 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900757 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900754,16900755,16900756,16900757,16900758' parsl.providers.slurm.slurm DEBUG: squeue returned 16900755 R 16900756 R 16900757 R 16900758 R 16900754 R parsl.providers.slurm.slurm DEBUG: Updating job 16900755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900754 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:11 proc_lsst.multi:146 [INFO] found job 16900758 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900758 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900754,16900755,16900756,16900757,16900758' parsl.providers.slurm.slurm DEBUG: squeue returned 16900755 R 16900756 R 16900757 R 16900758 R 16900754 R parsl.providers.slurm.slurm DEBUG: Updating job 16900755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900754 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:11 proc_lsst.multi:146 [INFO] found job 16900759 in provider escience_a40 proc_lsst.multi INFO: found job 16900759 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:11 proc_lsst.multi:146 [INFO] found job 16900760 in provider escience_a40 proc_lsst.multi INFO: found job 16900760 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:12 proc_lsst.multi:146 [INFO] found job 16900761 in provider escience_a40 proc_lsst.multi INFO: found job 16900761 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:12 proc_lsst.multi:146 [INFO] found job 16900762 in provider escience_a40 proc_lsst.multi INFO: found job 16900762 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:12 proc_lsst.multi:146 [INFO] found job 16900763 in provider escience_a40 proc_lsst.multi INFO: found job 16900763 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:12 proc_lsst.multi:146 [INFO] found job 16900764 in provider escience_a40 proc_lsst.multi INFO: found job 16900764 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:12 proc_lsst.multi:146 [INFO] found job 16900765 in provider escience_a40 proc_lsst.multi INFO: found job 16900765 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:13 proc_lsst.multi:146 [INFO] found job 16900766 in provider escience_a40 proc_lsst.multi INFO: found job 16900766 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:13 proc_lsst.multi:146 [INFO] found job 16900767 in provider escience_a40 proc_lsst.multi INFO: found job 16900767 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING 2024-03-07 19:12:13 proc_lsst.multi:146 [INFO] found job 16900768 in provider escience_a40 proc_lsst.multi INFO: found job 16900768 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16900759,16900760,16900761,16900762,16900763,16900764,16900765,16900766,16900767,16900768' parsl.providers.slurm.slurm DEBUG: squeue returned 16900765 R 16900766 R 16900767 R 16900768 R 16900763 R 16900764 R 16900760 R 16900761 R 16900762 R 16900759 R parsl.providers.slurm.slurm DEBUG: Updating job 16900765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16900759 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 26/20 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (46) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 3.337860107421875e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22886247474768 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=46, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 26 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3f50f709a2db parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3f50f709a2db parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6eff3e942c45 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6eff3e942c45 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1f322a0fa31c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1f322a0fa31c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b4e7c4073830 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b4e7c4073830 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 833e8fca381e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 833e8fca381e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 19e6604f795b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 19e6604f795b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8998c3e7d656 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8998c3e7d656 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 273f6f5147b9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 273f6f5147b9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c0b84c781f19 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c0b84c781f19 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6f0dae97b124 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6f0dae97b124 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff849a5f3d98 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff849a5f3d98 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 250f59b3df78 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 250f59b3df78 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 56e153dd52cf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 56e153dd52cf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e07c9e3085dd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e07c9e3085dd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8fbc08bbf03d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8fbc08bbf03d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b46ad90fe6ca parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b46ad90fe6ca parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6972661c72a3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6972661c72a3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1644a534d104 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1644a534d104 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e894f4feea05 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e894f4feea05 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 94ee92a02661 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 94ee92a02661 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 28c7fb408b03 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 28c7fb408b03 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c60db19d813d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c60db19d813d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b95b84441b14 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b95b84441b14 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dc5babc3fcb9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dc5babc3fcb9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b7648ef3f190 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b7648ef3f190 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d802de6fab65 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d802de6fab65 2024-03-07 19:12:13 proc_lsst.multi:146 [INFO] found job 16900730 in provider astro proc_lsst.multi INFO: found job 16900730 in provider astro 2024-03-07 19:12:13 proc_lsst.multi:201 [INFO] cancelling 16900730 on provider astro proc_lsst.multi INFO: cancelling 16900730 on provider astro 2024-03-07 19:12:13 proc_lsst.multi:146 [INFO] found job 16900731 in provider astro proc_lsst.multi INFO: found job 16900731 in provider astro 2024-03-07 19:12:13 proc_lsst.multi:201 [INFO] cancelling 16900731 on provider astro proc_lsst.multi INFO: cancelling 16900731 on provider astro 2024-03-07 19:12:14 proc_lsst.multi:146 [INFO] found job 16900729 in provider astro proc_lsst.multi INFO: found job 16900729 in provider astro 2024-03-07 19:12:14 proc_lsst.multi:201 [INFO] cancelling 16900729 on provider astro proc_lsst.multi INFO: cancelling 16900729 on provider astro 2024-03-07 19:12:14 proc_lsst.multi:146 [INFO] found job 16900732 in provider astro proc_lsst.multi INFO: found job 16900732 in provider astro 2024-03-07 19:12:14 proc_lsst.multi:201 [INFO] cancelling 16900732 on provider astro proc_lsst.multi INFO: cancelling 16900732 on provider astro 2024-03-07 19:12:14 proc_lsst.multi:146 [INFO] found job 42497 in provider local proc_lsst.multi INFO: found job 42497 in provider local 2024-03-07 19:12:14 proc_lsst.multi:201 [INFO] cancelling 42497 on provider local proc_lsst.multi INFO: cancelling 42497 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 42497 2024-03-07 19:12:14 proc_lsst.multi:146 [INFO] found job 16900725 in provider astro proc_lsst.multi INFO: found job 16900725 in provider astro 2024-03-07 19:12:14 proc_lsst.multi:201 [INFO] cancelling 16900725 on provider astro proc_lsst.multi INFO: cancelling 16900725 on provider astro 2024-03-07 19:12:14 proc_lsst.multi:146 [INFO] found job 16900766 in provider escience_a40 proc_lsst.multi INFO: found job 16900766 in provider escience_a40 2024-03-07 19:12:14 proc_lsst.multi:201 [INFO] cancelling 16900766 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900766 on provider escience_a40 2024-03-07 19:12:14 proc_lsst.multi:146 [INFO] found job 16900765 in provider escience_a40 proc_lsst.multi INFO: found job 16900765 in provider escience_a40 2024-03-07 19:12:14 proc_lsst.multi:201 [INFO] cancelling 16900765 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900765 on provider escience_a40 2024-03-07 19:12:14 proc_lsst.multi:146 [INFO] found job 16900768 in provider escience_a40 proc_lsst.multi INFO: found job 16900768 in provider escience_a40 2024-03-07 19:12:14 proc_lsst.multi:201 [INFO] cancelling 16900768 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900768 on provider escience_a40 2024-03-07 19:12:15 proc_lsst.multi:146 [INFO] found job 16900760 in provider escience_a40 proc_lsst.multi INFO: found job 16900760 in provider escience_a40 2024-03-07 19:12:15 proc_lsst.multi:201 [INFO] cancelling 16900760 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900760 on provider escience_a40 2024-03-07 19:12:15 proc_lsst.multi:146 [INFO] found job 16900767 in provider escience_a40 proc_lsst.multi INFO: found job 16900767 in provider escience_a40 2024-03-07 19:12:15 proc_lsst.multi:201 [INFO] cancelling 16900767 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900767 on provider escience_a40 2024-03-07 19:12:15 proc_lsst.multi:146 [INFO] found job 16900761 in provider escience_a40 proc_lsst.multi INFO: found job 16900761 in provider escience_a40 2024-03-07 19:12:15 proc_lsst.multi:201 [INFO] cancelling 16900761 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900761 on provider escience_a40 2024-03-07 19:12:15 proc_lsst.multi:146 [INFO] found job 16900759 in provider escience_a40 proc_lsst.multi INFO: found job 16900759 in provider escience_a40 2024-03-07 19:12:15 proc_lsst.multi:201 [INFO] cancelling 16900759 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900759 on provider escience_a40 2024-03-07 19:12:15 proc_lsst.multi:146 [INFO] found job 16900726 in provider astro proc_lsst.multi INFO: found job 16900726 in provider astro 2024-03-07 19:12:15 proc_lsst.multi:201 [INFO] cancelling 16900726 on provider astro proc_lsst.multi INFO: cancelling 16900726 on provider astro 2024-03-07 19:12:15 proc_lsst.multi:146 [INFO] found job 16900724 in provider astro proc_lsst.multi INFO: found job 16900724 in provider astro 2024-03-07 19:12:15 proc_lsst.multi:201 [INFO] cancelling 16900724 on provider astro proc_lsst.multi INFO: cancelling 16900724 on provider astro 2024-03-07 19:12:16 proc_lsst.multi:146 [INFO] found job 16900756 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900756 in provider escience_rtx6k 2024-03-07 19:12:16 proc_lsst.multi:201 [INFO] cancelling 16900756 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16900756 on provider escience_rtx6k 2024-03-07 19:12:16 proc_lsst.multi:146 [INFO] found job 16900757 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900757 in provider escience_rtx6k 2024-03-07 19:12:16 proc_lsst.multi:201 [INFO] cancelling 16900757 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16900757 on provider escience_rtx6k 2024-03-07 19:12:16 proc_lsst.multi:146 [INFO] found job 16900755 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900755 in provider escience_rtx6k 2024-03-07 19:12:16 proc_lsst.multi:201 [INFO] cancelling 16900755 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16900755 on provider escience_rtx6k 2024-03-07 19:12:16 proc_lsst.multi:146 [INFO] found job 16900758 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900758 in provider escience_rtx6k 2024-03-07 19:12:16 proc_lsst.multi:201 [INFO] cancelling 16900758 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16900758 on provider escience_rtx6k 2024-03-07 19:12:16 proc_lsst.multi:146 [INFO] found job 16900727 in provider astro proc_lsst.multi INFO: found job 16900727 in provider astro 2024-03-07 19:12:16 proc_lsst.multi:201 [INFO] cancelling 16900727 on provider astro proc_lsst.multi INFO: cancelling 16900727 on provider astro 2024-03-07 19:12:17 proc_lsst.multi:146 [INFO] found job 16900764 in provider escience_a40 proc_lsst.multi INFO: found job 16900764 in provider escience_a40 2024-03-07 19:12:17 proc_lsst.multi:201 [INFO] cancelling 16900764 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900764 on provider escience_a40 2024-03-07 19:12:17 proc_lsst.multi:146 [INFO] found job 16900754 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900754 in provider escience_rtx6k 2024-03-07 19:12:17 proc_lsst.multi:201 [INFO] cancelling 16900754 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16900754 on provider escience_rtx6k 2024-03-07 19:12:17 proc_lsst.multi:146 [INFO] found job 16900763 in provider escience_a40 proc_lsst.multi INFO: found job 16900763 in provider escience_a40 2024-03-07 19:12:17 proc_lsst.multi:201 [INFO] cancelling 16900763 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900763 on provider escience_a40 2024-03-07 19:12:17 proc_lsst.multi:146 [INFO] found job 16900728 in provider astro proc_lsst.multi INFO: found job 16900728 in provider astro 2024-03-07 19:12:17 proc_lsst.multi:201 [INFO] cancelling 16900728 on provider astro proc_lsst.multi INFO: cancelling 16900728 on provider astro 2024-03-07 19:12:17 proc_lsst.multi:146 [INFO] found job 16900733 in provider astro proc_lsst.multi INFO: found job 16900733 in provider astro 2024-03-07 19:12:17 proc_lsst.multi:201 [INFO] cancelling 16900733 on provider astro proc_lsst.multi INFO: cancelling 16900733 on provider astro 2024-03-07 19:12:17 proc_lsst.multi:146 [INFO] found job 16900762 in provider escience_a40 proc_lsst.multi INFO: found job 16900762 in provider escience_a40 2024-03-07 19:12:17 proc_lsst.multi:201 [INFO] cancelling 16900762 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900762 on provider escience_a40 parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-07 19:12:18 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-07 19:12:18 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16900724', '16900725', '16900726', '16900727', '16900728', '16900729', '16900730', '16900731', '16900732', '16900733', '16900734', '16900735', '16900736', '16900737', '16900738', '16900739', '16900740', '16900741', '16900742', '16900743', '16900744', '16900745', '16900746', '16900747', '16900748', '16900749', '16900750', '16900751', '16900752', '16900753', '16900759', '16900760', '16900761', '16900762', '16900763', '16900764', '16900765', '16900766', '16900767', '16900768', '16900754', '16900755', '16900756', '16900757', '16900758', '42497'] proc_lsst.multi INFO: new jobs since last cancel ['16900724', '16900725', '16900726', '16900727', '16900728', '16900729', '16900730', '16900731', '16900732', '16900733', '16900734', '16900735', '16900736', '16900737', '16900738', '16900739', '16900740', '16900741', '16900742', '16900743', '16900744', '16900745', '16900746', '16900747', '16900748', '16900749', '16900750', '16900751', '16900752', '16900753', '16900759', '16900760', '16900761', '16900762', '16900763', '16900764', '16900765', '16900766', '16900767', '16900768', '16900754', '16900755', '16900756', '16900757', '16900758', '42497'] 2024-03-07 19:12:18 proc_lsst.multi:146 [INFO] found job 16900724 in provider astro proc_lsst.multi INFO: found job 16900724 in provider astro 2024-03-07 19:12:18 proc_lsst.multi:201 [INFO] cancelling 16900724 on provider astro proc_lsst.multi INFO: cancelling 16900724 on provider astro 2024-03-07 19:12:18 proc_lsst.multi:146 [INFO] found job 16900725 in provider astro proc_lsst.multi INFO: found job 16900725 in provider astro 2024-03-07 19:12:18 proc_lsst.multi:201 [INFO] cancelling 16900725 on provider astro proc_lsst.multi INFO: cancelling 16900725 on provider astro 2024-03-07 19:12:18 proc_lsst.multi:146 [INFO] found job 16900726 in provider astro proc_lsst.multi INFO: found job 16900726 in provider astro 2024-03-07 19:12:18 proc_lsst.multi:201 [INFO] cancelling 16900726 on provider astro proc_lsst.multi INFO: cancelling 16900726 on provider astro 2024-03-07 19:12:18 proc_lsst.multi:146 [INFO] found job 16900727 in provider astro proc_lsst.multi INFO: found job 16900727 in provider astro 2024-03-07 19:12:18 proc_lsst.multi:201 [INFO] cancelling 16900727 on provider astro proc_lsst.multi INFO: cancelling 16900727 on provider astro 2024-03-07 19:12:19 proc_lsst.multi:146 [INFO] found job 16900728 in provider astro proc_lsst.multi INFO: found job 16900728 in provider astro 2024-03-07 19:12:19 proc_lsst.multi:201 [INFO] cancelling 16900728 on provider astro proc_lsst.multi INFO: cancelling 16900728 on provider astro 2024-03-07 19:12:19 proc_lsst.multi:146 [INFO] found job 16900729 in provider astro proc_lsst.multi INFO: found job 16900729 in provider astro 2024-03-07 19:12:19 proc_lsst.multi:201 [INFO] cancelling 16900729 on provider astro proc_lsst.multi INFO: cancelling 16900729 on provider astro 2024-03-07 19:12:21 proc_lsst.multi:146 [INFO] found job 16900730 in provider astro proc_lsst.multi INFO: found job 16900730 in provider astro 2024-03-07 19:12:21 proc_lsst.multi:201 [INFO] cancelling 16900730 on provider astro proc_lsst.multi INFO: cancelling 16900730 on provider astro 2024-03-07 19:12:21 proc_lsst.multi:146 [INFO] found job 16900731 in provider astro proc_lsst.multi INFO: found job 16900731 in provider astro 2024-03-07 19:12:21 proc_lsst.multi:201 [INFO] cancelling 16900731 on provider astro proc_lsst.multi INFO: cancelling 16900731 on provider astro 2024-03-07 19:12:22 proc_lsst.multi:146 [INFO] found job 16900732 in provider astro proc_lsst.multi INFO: found job 16900732 in provider astro 2024-03-07 19:12:22 proc_lsst.multi:201 [INFO] cancelling 16900732 on provider astro proc_lsst.multi INFO: cancelling 16900732 on provider astro 2024-03-07 19:12:23 proc_lsst.multi:146 [INFO] found job 16900733 in provider astro proc_lsst.multi INFO: found job 16900733 in provider astro 2024-03-07 19:12:23 proc_lsst.multi:201 [INFO] cancelling 16900733 on provider astro proc_lsst.multi INFO: cancelling 16900733 on provider astro 2024-03-07 19:12:23 proc_lsst.multi:146 [INFO] found job 16900734 in provider astro proc_lsst.multi INFO: found job 16900734 in provider astro 2024-03-07 19:12:23 proc_lsst.multi:201 [INFO] cancelling 16900734 on provider astro proc_lsst.multi INFO: cancelling 16900734 on provider astro 2024-03-07 19:12:23 proc_lsst.multi:146 [INFO] found job 16900735 in provider astro proc_lsst.multi INFO: found job 16900735 in provider astro 2024-03-07 19:12:23 proc_lsst.multi:201 [INFO] cancelling 16900735 on provider astro proc_lsst.multi INFO: cancelling 16900735 on provider astro 2024-03-07 19:12:23 proc_lsst.multi:146 [INFO] found job 16900736 in provider astro proc_lsst.multi INFO: found job 16900736 in provider astro 2024-03-07 19:12:23 proc_lsst.multi:201 [INFO] cancelling 16900736 on provider astro proc_lsst.multi INFO: cancelling 16900736 on provider astro 2024-03-07 19:12:24 proc_lsst.multi:146 [INFO] found job 16900737 in provider astro proc_lsst.multi INFO: found job 16900737 in provider astro 2024-03-07 19:12:24 proc_lsst.multi:201 [INFO] cancelling 16900737 on provider astro proc_lsst.multi INFO: cancelling 16900737 on provider astro 2024-03-07 19:12:24 proc_lsst.multi:146 [INFO] found job 16900738 in provider astro proc_lsst.multi INFO: found job 16900738 in provider astro 2024-03-07 19:12:24 proc_lsst.multi:201 [INFO] cancelling 16900738 on provider astro proc_lsst.multi INFO: cancelling 16900738 on provider astro 2024-03-07 19:12:24 proc_lsst.multi:146 [INFO] found job 16900739 in provider astro proc_lsst.multi INFO: found job 16900739 in provider astro 2024-03-07 19:12:24 proc_lsst.multi:201 [INFO] cancelling 16900739 on provider astro proc_lsst.multi INFO: cancelling 16900739 on provider astro 2024-03-07 19:12:24 proc_lsst.multi:146 [INFO] found job 16900740 in provider astro proc_lsst.multi INFO: found job 16900740 in provider astro 2024-03-07 19:12:24 proc_lsst.multi:201 [INFO] cancelling 16900740 on provider astro proc_lsst.multi INFO: cancelling 16900740 on provider astro 2024-03-07 19:12:24 proc_lsst.multi:146 [INFO] found job 16900741 in provider astro proc_lsst.multi INFO: found job 16900741 in provider astro 2024-03-07 19:12:24 proc_lsst.multi:201 [INFO] cancelling 16900741 on provider astro proc_lsst.multi INFO: cancelling 16900741 on provider astro 2024-03-07 19:12:25 proc_lsst.multi:146 [INFO] found job 16900742 in provider astro proc_lsst.multi INFO: found job 16900742 in provider astro 2024-03-07 19:12:25 proc_lsst.multi:201 [INFO] cancelling 16900742 on provider astro proc_lsst.multi INFO: cancelling 16900742 on provider astro 2024-03-07 19:12:25 proc_lsst.multi:146 [INFO] found job 16900743 in provider astro proc_lsst.multi INFO: found job 16900743 in provider astro 2024-03-07 19:12:25 proc_lsst.multi:201 [INFO] cancelling 16900743 on provider astro proc_lsst.multi INFO: cancelling 16900743 on provider astro 2024-03-07 19:12:25 proc_lsst.multi:146 [INFO] found job 16900744 in provider astro proc_lsst.multi INFO: found job 16900744 in provider astro 2024-03-07 19:12:25 proc_lsst.multi:201 [INFO] cancelling 16900744 on provider astro proc_lsst.multi INFO: cancelling 16900744 on provider astro 2024-03-07 19:12:25 proc_lsst.multi:146 [INFO] found job 16900745 in provider astro proc_lsst.multi INFO: found job 16900745 in provider astro 2024-03-07 19:12:25 proc_lsst.multi:201 [INFO] cancelling 16900745 on provider astro proc_lsst.multi INFO: cancelling 16900745 on provider astro 2024-03-07 19:12:25 proc_lsst.multi:146 [INFO] found job 16900746 in provider astro proc_lsst.multi INFO: found job 16900746 in provider astro 2024-03-07 19:12:25 proc_lsst.multi:201 [INFO] cancelling 16900746 on provider astro proc_lsst.multi INFO: cancelling 16900746 on provider astro 2024-03-07 19:12:26 proc_lsst.multi:146 [INFO] found job 16900747 in provider astro proc_lsst.multi INFO: found job 16900747 in provider astro 2024-03-07 19:12:26 proc_lsst.multi:201 [INFO] cancelling 16900747 on provider astro proc_lsst.multi INFO: cancelling 16900747 on provider astro 2024-03-07 19:12:26 proc_lsst.multi:146 [INFO] found job 16900748 in provider astro proc_lsst.multi INFO: found job 16900748 in provider astro 2024-03-07 19:12:26 proc_lsst.multi:201 [INFO] cancelling 16900748 on provider astro proc_lsst.multi INFO: cancelling 16900748 on provider astro 2024-03-07 19:12:26 proc_lsst.multi:146 [INFO] found job 16900749 in provider astro proc_lsst.multi INFO: found job 16900749 in provider astro 2024-03-07 19:12:26 proc_lsst.multi:201 [INFO] cancelling 16900749 on provider astro proc_lsst.multi INFO: cancelling 16900749 on provider astro 2024-03-07 19:12:26 proc_lsst.multi:146 [INFO] found job 16900750 in provider astro proc_lsst.multi INFO: found job 16900750 in provider astro 2024-03-07 19:12:26 proc_lsst.multi:201 [INFO] cancelling 16900750 on provider astro proc_lsst.multi INFO: cancelling 16900750 on provider astro 2024-03-07 19:12:26 proc_lsst.multi:146 [INFO] found job 16900751 in provider astro proc_lsst.multi INFO: found job 16900751 in provider astro 2024-03-07 19:12:26 proc_lsst.multi:201 [INFO] cancelling 16900751 on provider astro proc_lsst.multi INFO: cancelling 16900751 on provider astro 2024-03-07 19:12:26 proc_lsst.multi:146 [INFO] found job 16900752 in provider astro proc_lsst.multi INFO: found job 16900752 in provider astro 2024-03-07 19:12:26 proc_lsst.multi:201 [INFO] cancelling 16900752 on provider astro proc_lsst.multi INFO: cancelling 16900752 on provider astro 2024-03-07 19:12:27 proc_lsst.multi:146 [INFO] found job 16900753 in provider astro proc_lsst.multi INFO: found job 16900753 in provider astro 2024-03-07 19:12:27 proc_lsst.multi:201 [INFO] cancelling 16900753 on provider astro proc_lsst.multi INFO: cancelling 16900753 on provider astro 2024-03-07 19:12:27 proc_lsst.multi:146 [INFO] found job 16900759 in provider escience_a40 proc_lsst.multi INFO: found job 16900759 in provider escience_a40 2024-03-07 19:12:27 proc_lsst.multi:201 [INFO] cancelling 16900759 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900759 on provider escience_a40 2024-03-07 19:12:27 proc_lsst.multi:146 [INFO] found job 16900760 in provider escience_a40 proc_lsst.multi INFO: found job 16900760 in provider escience_a40 2024-03-07 19:12:27 proc_lsst.multi:201 [INFO] cancelling 16900760 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900760 on provider escience_a40 2024-03-07 19:12:27 proc_lsst.multi:146 [INFO] found job 16900761 in provider escience_a40 proc_lsst.multi INFO: found job 16900761 in provider escience_a40 2024-03-07 19:12:27 proc_lsst.multi:201 [INFO] cancelling 16900761 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900761 on provider escience_a40 2024-03-07 19:12:27 proc_lsst.multi:146 [INFO] found job 16900762 in provider escience_a40 proc_lsst.multi INFO: found job 16900762 in provider escience_a40 2024-03-07 19:12:27 proc_lsst.multi:201 [INFO] cancelling 16900762 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900762 on provider escience_a40 2024-03-07 19:12:28 proc_lsst.multi:146 [INFO] found job 16900763 in provider escience_a40 proc_lsst.multi INFO: found job 16900763 in provider escience_a40 2024-03-07 19:12:28 proc_lsst.multi:201 [INFO] cancelling 16900763 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900763 on provider escience_a40 2024-03-07 19:12:28 proc_lsst.multi:146 [INFO] found job 16900764 in provider escience_a40 proc_lsst.multi INFO: found job 16900764 in provider escience_a40 2024-03-07 19:12:28 proc_lsst.multi:201 [INFO] cancelling 16900764 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900764 on provider escience_a40 2024-03-07 19:12:28 proc_lsst.multi:146 [INFO] found job 16900765 in provider escience_a40 proc_lsst.multi INFO: found job 16900765 in provider escience_a40 2024-03-07 19:12:28 proc_lsst.multi:201 [INFO] cancelling 16900765 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900765 on provider escience_a40 2024-03-07 19:12:28 proc_lsst.multi:146 [INFO] found job 16900766 in provider escience_a40 proc_lsst.multi INFO: found job 16900766 in provider escience_a40 2024-03-07 19:12:28 proc_lsst.multi:201 [INFO] cancelling 16900766 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900766 on provider escience_a40 2024-03-07 19:12:28 proc_lsst.multi:146 [INFO] found job 16900767 in provider escience_a40 proc_lsst.multi INFO: found job 16900767 in provider escience_a40 2024-03-07 19:12:28 proc_lsst.multi:201 [INFO] cancelling 16900767 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900767 on provider escience_a40 2024-03-07 19:12:29 proc_lsst.multi:146 [INFO] found job 16900768 in provider escience_a40 proc_lsst.multi INFO: found job 16900768 in provider escience_a40 2024-03-07 19:12:29 proc_lsst.multi:201 [INFO] cancelling 16900768 on provider escience_a40 proc_lsst.multi INFO: cancelling 16900768 on provider escience_a40 2024-03-07 19:12:29 proc_lsst.multi:146 [INFO] found job 16900754 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900754 in provider escience_rtx6k 2024-03-07 19:12:29 proc_lsst.multi:201 [INFO] cancelling 16900754 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16900754 on provider escience_rtx6k 2024-03-07 19:12:29 proc_lsst.multi:146 [INFO] found job 16900755 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900755 in provider escience_rtx6k 2024-03-07 19:12:29 proc_lsst.multi:201 [INFO] cancelling 16900755 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16900755 on provider escience_rtx6k 2024-03-07 19:12:29 proc_lsst.multi:146 [INFO] found job 16900756 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900756 in provider escience_rtx6k 2024-03-07 19:12:29 proc_lsst.multi:201 [INFO] cancelling 16900756 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16900756 on provider escience_rtx6k 2024-03-07 19:12:29 proc_lsst.multi:146 [INFO] found job 16900757 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900757 in provider escience_rtx6k 2024-03-07 19:12:29 proc_lsst.multi:201 [INFO] cancelling 16900757 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16900757 on provider escience_rtx6k 2024-03-07 19:12:29 proc_lsst.multi:146 [INFO] found job 16900758 in provider escience_rtx6k proc_lsst.multi INFO: found job 16900758 in provider escience_rtx6k 2024-03-07 19:12:29 proc_lsst.multi:201 [INFO] cancelling 16900758 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16900758 on provider escience_rtx6k 2024-03-07 19:12:30 proc_lsst.multi:146 [INFO] found job 42497 in provider local proc_lsst.multi INFO: found job 42497 in provider local 2024-03-07 19:12:30 proc_lsst.multi:201 [INFO] cancelling 42497 on provider local proc_lsst.multi INFO: cancelling 42497 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 42497 parsl.providers.local.local WARNING: Failed to kill PID: 42497 and child processes on local 2024-03-07 19:12:31 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 119.8919 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20220827_B1l_science#step0_20240307T191019Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 119.8987 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 130.2507 seconds; current memory usage: 0.272 Gibyte, delta: 0.135 Gibyte, peak delta: 0.137 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.274 Gibyte (largest child process) Run Id: None Run Name: DEEP_20220827_B1l_science#step0_20240307T191019Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up